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Preface 


It  is  a  common  generalisation  that  it  is  impractical  to  experiment  < 

with  large  complex  real  life  systems  such  as  airbases  to  determine  how  they 
are  likely  to  perform  under  a  range  of  conditions.  Simulation  models  are  r 

often  called  upon  to  represent  as  faithfully  as  possible  how  the  real  system 
responds  to  various  sets  of  input  conditions,  but  as  the  system  becomes 
more  complex,  so  too  does  the  simulation  model.  A  point  may  be  reached  at 
which  the  simulation  model  itself  is  either  too  complex,  or  too  resource 
intensive  to  be  practically  used  for  day  to  day  analysis.  At  this  point  we 
turn  to  a  second  level  of  modeling,  and  produce  an  analytic  model  of  the 
simulation  input-output  relationship,  known  as  a  metamodel. 

This  study  follows  previous  research  by  Lt  Col  David  A.  Diener, 

USAF,  who  used  a  simulation  model  to  represent  the  sortie  generation 
capability  of  an  airbase,  and  then  developed  metamodels  from  the 
simulation  results.  This  study  does  not  address  the  simulation  issues,  but 
examines  instead  the  possibility  of  finding  alternate  metamodels  to 
represent  the  behaviour  of  the  airbase  system.  Several  aspects  of  the  model 
development  process  are  also  examined. 

Several  people  deserve  recognition  for  the  completion  of  this  project.  * 

Lt  Col  Diener  provided  the  germ  of  an  idea  for  this  research,  made  his  data 
available,  and  was  always  ready  to  provide  guidance  and  assistance  as  the 
research  progressed.  He  also  instilled  in  me  an  ongoing  interest  in 


11 


simulation,  and  for  his  contribution  to  both  my  education  and  this  project,  I 
am  very  grateful.  I  also  wish  to  thank  Lt  Col  Phillip  E.  Miller,  USAF  for 
his  insights  and  assistance  with  this  research,  particularly  in  providing 
direction  and  encouragement  when  the  path  ahead  was  not  at  all  clear. 

My  deepest  thanks,  though,  are  reserved  for  my  family,  particularly 
my  wife  Kim,  who,  far  from  home  and  with  a  new  baby,  gave  me  all  the 
love,  support  and  encouragement  I  could  have  asked,  and  certainly  more 
than  I  deserved.  Without  her  holding  our  family  together  while  I  worked, 
this  thesis  would  not  have  been  completed. 

I  wish  to  dedicate  this  work  to  my  children,  Lachlan  and  Cameron,  in 
the  hope  that  in  their  world  of  the  future,  the  things  that  we  research  now 
are  never  tested. 


Alistair  Dally 
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Abstract 

m 

This  study  is  an  exploratory  investigation  into  the  development  of 
metamodels  of  a  particular  Theater  Simulation  of  Airbase  Resources  (TSAR)  • 

simulation  model.  Techniques  applicable  to  the  development  of  metamodels 
from  a  simulation  using  an  orthogonal  two-level  fractional  factorial 
experimental  design  are  discussed.  The  experimental  design  is  found  to 
limit  the  metamodel  form  to  polynomial  linear  least  squares  regression 
models,  and  also  to  greatly  simplify  the  process  of  building  regression 
models.  Using  the  simulation  results  from  previous  research,  alternate 
metamodels  are  proposed  for  both  the  prediction  of  sorties  generated  by  the 
airbase  system,  as  modeled  by  the  simulation;  and  to  assist  in 
understanding  and  explaining  the  relationships  between  airbase  resource 
levels,  their  interactions,  and  sorties  generated.  The  metamodels  developed 
for  prediction  are  found  to  be  substantially  different  from  the  metamodels 
for  explanation.  While  unable  to  develop  metamodels  for  explanation 
significantly  different  from  the  metamodels  from  the  previous  research,  this 
research  confirms  that  the  existing  metamodels  are  the  best  possible  models 
that  meet  the  chosen  selection  criteria.  Several  lessons  pertaining  to  the  « 

metamodeling  process  for  large  scale  simulations  using  similar  experimental 
designs  are  proposed.  A  flexible  manual  alternative  to  computerised  model  * 

reduction  techniques  is  also  proposed. 
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AN  INVESTIGATION  OF  ALTERNATIVE  METAMODELS  FOR  THE 


THEATER  SIMULATION  OF  AIRBASE  RESOURCES  MODEL 


I.  Introduction 


1.1  Air  Base  Operability 

Air  base  operability  is  the  ability  of  an  air  base,  whether  under  attack 
or  not,  to  generate  aircraft  sorties  over  a  period  of  time  during  conflict. 

There  are  many  factors  which  influence  the  rate  at  which  sorties  can  be 
generated.  Some  of  the  major  factors  are  the  number  of  aircraft  on  the 
base;  the  availability  of  replacement  aircraft;  the  stocking  level  and 
replenishment  policies  for  fuel,  weapons  and  spare  parts;  the  availability  of 
maintenance  facilities,  including  specialised  test  equipment;  the  numbers  of 
support  personnel,  including  various  specialists;  the  availability  of  ground 
support  equipment;  and  the  level  of  attrition  the  aircraft  experience  (Diener, 
1989:2).  These  interactive  factors  may  be  analysed  in  a  scenario  where  the 
base  itself  either  comes  under  attack,  or  does  not.  It  is  not  feasible, 
however,  to  carry  out  the  desired  analysis  by  experimenting  directly  with 
such  a  large  scale  system,  so  we  turn  to  representing  the  behaviour  of  the 
system  with  some  form  of  model.  Large  scale  complex  systems  such  as  an 
airbase  are  often  portrayed  by  simulation  models. 


1.1.1  A  System  Simulation  Model.  Of  the  factors  listed  above,  all  but 


the  aircraft  attrition  rate  are  controllable  through  the  setting  of  resource 

and  logistics  policies.  Clearly,  determining  the  appropriate  levels  of  the 

factors  listed  is  a  difficult  and  complex  task,  particularly  when  the 

significant  potential  for  interaction  or  interdependency  between  many  of  the  » 

factors  is  considered.  Unfortunately,  many  studies  look  at  the  effect  on 

operability  of  only  one  variable  in  isolation.  The  Theater  Simulation  of 

Airbase  Resources  (TSAR)  model  is  a  simulation  model  designed  to  evaluate 

Air  Base  Operability  (ABO)  from  a  total  system  viewpoint  (Diener,  1989:3). 

However,  to  achieve  the  ability  to  take  a  total  system  viewpoint,  the  TSAR 
model  is  extremely  complicated.  To  illustrate  this  complexity,  when 
determining  aircraft  availability  on  an  F-15  base  the  model  considers  the 
possible  failure  of  81  aircraft  systems  and  subsystems,  as  well  as  including 
all  the  other  factors  mentioned  above.  (Diener,  1989:68). 

1.2  Previous.  Research 

In  his  1989  dissertation,  Diener  used  the  TSAR  model  to  simulate  the 
operability  of  a  European  F-15  base.  In  that  research,  he  used  a  one/eighth 
fractional  factorial  experimental  design  for  ten  factors  that  were  considered 
important  inputs  into  the  TSAR  model.  The  fractional  design  reduced  the  * 

number  of  treatments;  ie,  the  combinations  of  the  input  factors,  from  1024, 
which  would  be  required  for  a  full  factorial  design,  to  128.  Even  with  an  * 

efficient  experimental  design,  to  apply  all  128  treatments  to  each  of  30  days 


1.2 


operation  under  a  no-attack  and  an  attack  scenario  required  a  total  of  14 
hours  processing  time  on  a  Gould  NP-1  supercomputer  (Diener,  1989:72), 

»  plus  the  time  required  to  input  each  treatment.  Clearly,  the  cost  in  both 
time  and  money  of  making  multiple  runs  of  the  simulation  to  evaluate 
«  changes  in  individual  factors  is  very  high. 

1.3  Simulation  Disadvantages 

One  of  the  major  limitations  of  any  simulation  model  is  that  it  does 
not  provide  an  optimal  solution,  but  rather  evaluates  an  outcome  or 
response  only  for  the  particular  situation  represented  by  the  input  variables 
(Schriber,  1991:9).  Even  running  multiple  simulations  is  not  a  guarantee  of 
finding  an  optimal  solution:  the  best  that  can  be  said  is  that  a  good  solution 
has  been  found.  Related  to  the  problem  of  non-optimisation  is  the  foct  that 
the  results  from  a  simulation  run  generally  do  not  have  external  validity; 
that  is,  the  results  can  not  be  generalised  for  conditions  other  than  those 
specifically  simulated  (Schriber,  1991:9).  For  example,  Diener's  simulation 
applies  specifically  to  an  F-15  base  in  Europe,  and  his  results  should  not  be 
generalised  as  applicable  to,  say,  F-16  bases  in  the  United  States. 

»  1.4  Introduction  to  Metamndels 

The  limitations  of  a  simulation  model  described  above  can  be  partially 
«  overcome  by  developing  a  mathematical  model  of  the  simulation  response  to 
the  input  variables.  Within  the  range  of  conditions  simulated,  the 


1.3 


mathematical  model  allows  the  estimation  of  relationships  between  varying 
inputs  and  their  corresponding  outputs  which  may  not  have  been  actually 

« 

simulated.  Additionally,  at  least  over  a  limited  range,  the  mathematical 
model  may  allow  inferences  to  be  made  about  optimal  solutions  (Schriber, 
1991:9).  Such  a  model  is  known  as  a  metamodel.  Kleijnen  (1987:148)  and 
Sargent  (1991:888)  both  show  metamodels  in  a  hierarchical  relationship, 
with  a  simulation  model  as  a  representation  of  reality,  and  a  metamodel  as 
a  representation  of  the  simulation  model.  Kleijnen  articulates  the  concept 
of  a  metamodel  as  follows: 

From  the  "mess"  of  reality  we  proceed  to  a  well  structured  simulation 
model,  and  next  we  model  the  relationship  between  the  inputs  and 
outputs  of  this  simulation  model  with  a  regression  model.  (Kleijnen, 
1987:148) 

in  a  similar  definition,  Friedman  suggests  that: 

The  simulation  model,  although  simpler  than  the  real  world  system, 
is  still  a  very  complex  way  of  relating  input  to  output.  Since  one  of 
the  aims  of  most  simulations  must  be  to  gain  an  understanding  of 
this  relationship,  an  even  simpler  model  may  be  used  in  addition  to 
the  simulation  model.  When  a  model  is  used  as  a  device  to  better 
understand  and  explore  a  more  complex  model,  the  simpler,  auxiliary 
model  is  frequently  referred  to  as  a  metamodel.  (Friedman,  1983:28) 

A  more  succinct  definition  by  Kleijnen  is  that  the  metamodel  or  regression 

equation  is  "a  model  of  the  input/output  behavior  of  the  simulation 

computer  program"  (Kleijnen,  1992:1165). 

The  metamodel  is  developed  from  the  output  of  the  simulation  model 

using  regression  techniques,  and  once  validated,  may  be  used  to  analyse 
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and  predict  the  behaviour  of  the  simulation  model;  that  is,  "what  if* 
analysis,  among  other  things,  becomes  both  practical  and  inexpensive. 

1.4.1  Prediction  and  Explanation.  Although  the  terms  are  often  used 
together,  the  purposes  of  explanation  and  prediction  should  be 
distinguished.  The  two  purposes  are  not  mutually  exclusive,  but  different 
aspects  of  developing  the  regression  metamodel  may  need  to  be  more  or  less 
emphasised  depending  on  the  particular  purpose  for  which  the  metamodel  is 
intended.  'When  prediction  is  the  primary  purpose  of  the  model,  the  aim  is 
to  estimate  with  as  much  accuracy  and  precision  as  possible  the  likely  value 
of  the  response  to  a  given  set  of  values  for  the  independent  variables 
(Miller,  1990:2;  Neter  et  al.,  1989:436).  On  the  other  hand,  when 
explanation  is  the  primary  purpose,  the  emphasis  changes  towards 
discovering  which  of  the  variables  have  important  significant  effects  on  the 
response,  and  then  estimating  those  effects.  In  the  context  of  this  and 
Diener's  research,  an  explanatory  model  seeks  to  identify  a  relatively  small 
number  of  factors  that  have  the  most  impact  on  the  sortie  generation 
capability  of  the  airbase,  so  that  those  factors  can  be  emphasised  when 
resource  level  and  logistics  policies  are  set.  A  predictive  model,  on  the  other 
hand,  could  be  used  to  estimate  the  likely  capability  of  the  airbase  given  an 
existing  policy.  Because  an  existing  resource  profile  represents  only  one 
alternative  to  enumerate,  the  predictive  model  can  be  more  complex  without 
losing  its  usefulness.  The  issue  of  how  many  variables  to  include  in  both 
predictive  and  explanatory  models  will  be  further  addressed  in  Chapter  III. 
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1.5  Easting  Metamodels 

The  most  common  form  of  metamodel  is  a  linear  regression  model. 
Diener  (1989)  developed  linear  regression  models  to  estimate  the  response  of 
the  TSAR  simulation  model  for  each  day  of  a  thirty-day  period.  Each  model 
potentially  consists  of  63  terms,  comprising  the  constant  or  intercept  term,  a 
term  for  each  of  the  ten  main  factors,  and  a  term  for  each  of  the  45  two-way 
interactions  between  the  main  factors,  and  (included  in  all  models)  seven 
terms  which  represent  the  effect  of  the  blocking  used  in  the  experimental 
design.  One  metamodel  was  calculated  for  each  day  of  the  period  simulated, 
for  both  the  attack  and  no-attack  cases,  resulting  in  a  total  of  sixty 
metamodels.  Each  metamodel  is  an  analytic  representation  of  the  ability  of 
the  airbase  to  generate  sorties  on  that  day,  given  the  level  of  the  input 
factors  that  existed  on  the  first  day.  The  model  reduction  technique  used 
resulted  in  metamodels  generally  containing  about  20  terms  (excluding  the 
blocking  terms),  with  the  remainder  of  the  terms  insignificant  at  the  0.10 
level. 

1.6  Problem  Statement 

The  large  number  of  terms  remaining  in  the  daily  metamodels  makes 
meaningful  analysis  difficult,  particularly  if  the  aim  is  to  explain  the 
behaviour  of  the  model  as  distinct  from  predicting  the  number  of  sorties  on 
a  given  day.  Additionally,  although  the  linear  regression  models  developed 
were  valuable  in  understanding  and  predicting  the  behaviour  of  the 
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simulation,  significant  amounts  of  the  total  variance  in  the  response  were 
unexplained.  Finally,  the  behaviour  of  the  airbase  over  time  is  not 
►  explicitly  represented,  because  each  metamodel  represents  only  a  daily 

snapshot  of  the  airbase  performance. 

* 

1.7  Research  Questions 

The  primary  question  to  be  answered  in  this  research  is  whether 
alternative  metamodels  can  be  developed  to  assist  in  the  analysis  and 
prediction  of  the  response  of  the  TSAR  simulation  as  carried  out  by  Diener 
in  his  1989  doctoral  research. 

A  secondary  question  to  be  answered  is  whether  alternative 
approaches  to  the  experimental  design  used  by  Diener  could  facilitate  the 
development  of  useful  metamodels. 

1.8  Research  Objectives 

The  primary  objective  of  this  research  is  to  investigate  whether 
alternative  metamodels  other  than  those  derived  by  Diener  can  be  used  to 
effectively  represent  the  results  of  Diener's  simulation.  To  achieve  this 
objective  the  following  questions  require  answers: 

1.  What  is  the  purpose  of  the  metamodel?  For  example,  is 

understanding  general  relationships  in  the  system  as  simulated  the 

r> 

primary  goal,  or  do  we  wish  to  make  predictions  about  the  response  of 
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the  simulation  under  different  conditions?  Do  different  goals  require 
different  models? 

2.  What  are  the  important  criteria  in  determining  the  suitability  of  a  ^ 

metamodel?  For  example,  is  the  overall  fit  of  the  model  the  primary 
criterion,  or  are  there  other  important  factors  to  consider?  , 

3.  How  does  the  nature  of  the  output  data,  and  the  experimental  design 
on  which  it  is  based,  limit  or  restrict  the  types  of  metamodels  that 
can  be  developed? 

1.9  Scope 

The  scope  of  this  research  is  specifically  limited  by  the  existing 
database  developed  by  Diener  during  his  doctoral  research.  The  database 
comprises  the  design  matrix  for  the  experimental  design,  which  allowed 
estimation  of  ten  main  effects  and  their  two-way  interactions,  and  the  thirty 
daily  simulated  responses  to  the  design  inputs  for  both  the  attack  and  no¬ 
attack  cases. 

1.1Q..  Jtesgarch  Plan 

In  Chapter  n,  a  more  detailed  treatment  of  the  background  to  the 
present  study  is  presented.  A  review  of  the  literature  relating  to  « 

metamodeling  and  the  relevant  aspects  of  simulation  and  experimental 
design  is  also  presented.  ' 
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Chapter  HI  describes  the  investigation  of  the  current  metamodels, 
and  the  exploration  of  techniques  to  develop  possible  new  models  and  model 
forms.  Chapter  IV  summarises  the  lessons  learned  from  the  exploration, 
presents  conclusions,  recommendations,  and  possible  directions  for  further 
research. 

Ill  Summary 

This  chapter  provided  a  brief  introduction  to  the  simulation  of 
Airbase  Operability  carried  out  in  previous  research  by  Diener.  The  concept 
of  metamodels  as  an  adjunct  to  simulation  analysis  was  also  introduced. 
Both  the  previous  research  and  simulation  metamodels  in  general  are 
presented  in  more  detail  in  the  next  chapter. 


II.  Background  and  Literature  Review 


r2.1  Introduction 

As  this  study  relies  on  the  results  of  previous  research,  the  first  part 
♦of  this  chapter  examines  some  of  the  key  points  of  that  research.  The  key 
points  relate  to  the  experimental  design  used  previously,  which  determined 
the  data  available  for  this  study,  and  the  form  of  the  metamodels  developed 
by  Diener.  Taking  a  more  general  view,  some  applications  of  simulation 
metamodels  and  some  of  the  current  research  issues  in  simulation 
metamodeling  are  discussed,  followed  by  a  review  of  some  of  the  literature 
pertaining  to  regression  and  regression  model  building  techniques. 


2.2  Previous  Research  Kev  Points 

2.2.1  Diener’a  Research  Objectives.  The  research  objectives  of 
Diener's  study  were  as  follows: 

1)  Efficiently  apply  an  experimental  design  that  will  reduce 
variance  due  to  the  inherent  randomness  of  the  TSAR  and 
TSARINA  simulation  models; 

2)  Estimate  metamodels,  with  significant  main  effects  and  two- 
way  interactions,  from  large  scale  simulation  experiments  so 
that  sorties  flown  can  be  predicted  based  on  input  factors; 

►  3)  Evaluate  the  impact  of  air  base  attacks  on  sorties  flown;  and 

4)  Identify  key  resources  and/or  interactions  over  a  thirty-day 

time  period  with  and  without  air  base  attacks.  (Diener,  1989:9) 
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This  study  is  most  closely  aligned  with  the  second  and  fourth  of  Diener's 
objectives,  although  the  experimental  design  resulting  from  Diener's  first 
objective  dictates  the  form  of  the  data  available  for  this  research.  The  t 

metamodels  estimated  by  Diener  in  achieving  his  second  objective  provide 
the  baseline  for  the  comparison  of  any  alternate  models  developed.  If  , 

alternate  models  can  be  developed,  their  interpretation  may  lead  to  the 
identification  of  key  resources  different  from  those  identified  as  Diener's 
fourth  objective. 

2.2.2  Experimental  Design.  Diener  assumed  that  higher  than  two- 
way  interactions  were  insignificant,  and  therefore  chose  a  1/8  fractional- 
factorial  experimental  design  (210"3  Resolution  V),  resulting  in  128  input 
combinations  or  treatments  (Diener,  1989:44).  The  1/8  fractional-factorial 
design  is  able  to  measure  the  individual,  or  main,  effect  of  each  variable  and 
all  their  two-way  interactions  without  confounding  between  those  effects. 

The  effects  of  possible  third  and  higher  order  interactions  cannot  be 

discriminated,  and  if  present,  will  appear  to  contribute  to  the  residual 

variance.  To  isolate  the  effect  of  attacks  on  the  base,  each  treatment  was 

applied  in  both  the  no-attack  and  the  attack  case,  thus  requiring  a  total  of 

256  different  simulation  runs  (Diener,  1989:46).  It  is  also  important  to  note 

that  each  of  the  128  design  points  or  treatments  evaluated  represents  «, 

specific  resource  levels  and  logistics  policies  (eg.  fuel  resupply  schedule)  in 

effect  at  the  airbase  on  the  first  day  of  the  simulation.  * 
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In  an  effort  to  reduce  the 


2.2.3  Reduction  of  Variance  and  Blocking, 
variance  in  the  simulation  output,  Diener  used  blocking  based  on  pseudo¬ 
random  number  streams.  Blocking  in  this  case  is  the  grouping  of  several 
treatments  together,  and  then  running  the  simulation  with  the  same 
starting  point  of  the  pseudo-random  number  stream  for  each  treatment 
within  the  block  (Diener,  1989:46-47).  For  the  attack  case,  one  specific 
version  of  the  TSARINA  attacks  was  applied  within  each  block.  The  128 
design  points  in  the  experiment  were  divided  into  eight  blocks  of  sixteen 
treatments  each.  The  effect  of  the  blocking  on  the  experimental  results  is 
that  block  effect  terms  must  be  included  in  all  the  metamodels,  as  outlined 
in  the  next  section. 

2.2.4  Existing  Metamodels.  The  form  of  metamodel  chosen  by  Diener 
to  represent  the  simulation  output  was  a  multiple  linear  regression  model 
which  considered  the  main  effects  and  two-way  interactions  of  the  factors, 
and  the  effect  of  the  blocking.  The  models  were  calculated  from  the 
simulation  results  using  ordinary  least  squares  regression.  For  the  no¬ 
attack  case,  the  metamodels  are: 

S,  =  P/i)  +  Bud)  +  Zj  V/VXjj  +  Ij  VjtttJX^  +  eft)  (2.1) 

where  S,  is  the  number  of  sorties  generated  on  day  i; 

Xy  and  Xlk  are  the  level  of  factor  j  and  k  in  effect  on  the  first  day,  for 
j=0,..,9,  k=j+l; 
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Bn(i)  reflects  the  random  effect  on  day  i  due  to  the  random  number  streams 
in  TSAR,  (block  effects)  where  Bn(i)  -  N(0,ajj);  and  e(i)  reflects  the 
experimental  error,  where  e(i)  -  N(0,o^). 

For  the  attack  case,  the  metamodels  are: 

$  =  $J(i)  +  B'n(i)  +  Z,  fr/OXy  +  2^  X*  PjJOXvX,,,  +  E(i)  (2.2) 

where  S*  is  the  number  of  sorties  generated  on  day  i, 

Xjj  and  X*  are  the  level  of  factor  j  and  k  in  effect  on  the  first  day,  for 
j=0,..,9,  k=j+l; 

B^(i)  reflects  the  random  effect  on  day  i  due  to  the  random  number  streams 
in  TSAR  and  TSARINA,  (block  effects)  where  Bj2(i)  ~  N(0,a|«);  and 
e*(i)  reflects  the  experimental  error,  where  e*(i)  -  N(0,o^*). 

The  eventual  metamodels  include  only  the  terms  that  have  a  significant 
effect,  with  the  exception  of  the  blocking  terms  which  are  included  in  all 
models  regardless  of  significance  (Diener,  1989:50-51). 

In  interpreting  the  no-attack  case  metamodels,  the  intercept 
parameters  Po(i)  represent  the  number  of  sorties  flown  on  the  ith  day  when 
all  factors  are  at  their  low  level  on  day  one;  the  main  effect  parameters  P/i) 
represent  the  change  in  the  number  of  sorties  flown  on  the  ith  day  when  the 
jth  factor  is  changed  from  its  low  level  on  day  one  to  its  high  level;  and  the 
interaction  effect  parameters  Pj1t(i)  represent  the  change  in  the  number  of 
sorties  flown  on  the  ith  day  when  the  interaction  term  changes  from  a  low 
to  a  high  level  on  day  one.  The  same  analysis  holds  for  the  starred 
parameters  in  the  attack  case  (Diener,  1989:75,102). 
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2.2.5  Design  Coding.  In  developing  the  design  matrix  for  his 
analysis,  Diener  originally  coded  the  low  level  of  factor  Xj  as  0,  and  the  high 
♦level  as  1  but  has  since  reaccomplished  the  analysis  with  low  factor  levels 
coded  as  -1  and  high  levels  as  1.  Many  regression  texts  refer  to  (0,1)  coding 

'for  two  level  qualitative  variables,  but  several  authors  including  Kleijnen  et 
al.  (1979:53),  Smith  and  Mauro  (1984:254),  and  Kleijnen  (1992:1165) 
suggest  that  qualitative  variables  should  be  coded  as  (-1,1).  This  research 
will  adopt  the  (-1,1)  coding  for  further  metamodel  development  and 
analysis. 

2.2.6  Assumptions  Required.  The  metamodels  above  require  three 
major  assumptions  to  be  made.  The  first  assumption  is  that  there  is  a 
linear  relationship  between  the  level  of  the  factors  and  the  response,  the 
number  of  sorties  flown.  The  second  assumption,  a  requirement  of  using 
linear  least  squares  technique,  is  that  for  each  model  the  variance  of  the 
error  term  is  constant  for  all  values  of  the  response,  i.e.  homoscedasticity 
e  asts.  The  final  assumption  is  that  the  error  terms  must  be  normally 
distributed  (Neter  et  al.,  1989:Ch  4).  For  the  first,  fifth,  and  last  day  of  the 
simulation,  Diener  tested  his  results  for  homoscedasticity  both  within  the 
attack  and  no-attack  cases  and  between  the  cases.  Within  each  case,  the 

*  hypothesis  of  homoscedasticity  was  not  rejected  at  the  a=0.05  level,  but  was 
rejected  when  comparing  between  cases  (Diener,  1989:56-58;  86).  Analysis 

*  of  residuals  using  stem-and-leaf  plots,  box  plots,  and  normal  probability 
plots  suggested  that  assuming  normality  in  the  error  terms  was  also 
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reasonable  (Diener,  1989iAppendices  C  and  D).  No  analysis  of  residuals 

against  the  dependent  or  independent  variables  was  carried  out.  The 

validity  of  these  assumptions  is  further  discussed  in  the  next  chapter.  , 

2.3  Simulation  Metamodeling  Literature  * 

The  literature  dealing  with  simulation  metamodeling  can  be  readily 
divided  into  two  categories.  The  first  category  reports  applications  of 
simulation  metamodeling,  while  the  second  deals  with  research  issues  and 
theoretical  aspects.  Neither  category  could  be  described  as  extensive,  and 
no  literature  that  relates  closely  to  the  current  study  has  been  found.  This 
research  is  atypical  for  two  reasons.  First,  it  has  many  independent 
variables,  all  of  which  are  qualitative  while  other  studies  contain  fewer 
quantitative  variables.  Second,  the  TSAR  model  is  a  highly  complex 
logistics  system  model,  while  most  reported  applications  of  metamodeling 
deal  with  some  variation  of  a  queuing  problem.  Possibly  the  most  unusual 
feature  of  this  research,  however,  is  the  generation  of  a  time  series  of  30 
daily  responses  for  each  input  treatment. 

The  regression  literature  is  also  of  little  help,  as  most  authors 
concentrate  on  observational  rather  than  designed  experiments,  with 
apparently  little  emphasis  on  models  as  complex  as  in  this  research.  The  «. 

lack  of  applicability  to  this  research  of  the  regression  and  model  building 
techniques  used  in  observational  experiments  are  further  discussed  in  the  * 

next  chapter. 
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Several  reported  applications  of  simulation  metamodeling  are 
reviewed,  followed  by  a  review  of  some  current  research  issues  in  this  field. 

2.3.1  Flexible  Manufacturing  System  Case  Study.  Kleijnen  and 
Standridge  (1988:257-261)  describe  a  case  study  of  a  flexible  manufacturing 
system,  concentrating  on  the  issues  of  experimental  design,  and  the  form  of 
the  metamodel  chosen.  Although  their  study  involved  a  deterministic 
simulation,  and  only  four  factors  were  involved,  several  important  points 
are  made.  First,  experimental  design  and  therefore  the  input  combinations 
modeled  determine  the  possible  form  of  a  metamodel.  For  example,  if 
interactive  effects  are  assumed  to  be  non-existent,  fewer  input  combinations 
are  required  than  if  the  interactions  are  present  (Kleijnen  and  Standridge, 
1988:259).  The  reverse  case  is  also  true,  and  affects  the  present  research  in 
that  the  experimental  design  used  limits  the  analysis  to  main  effects  and 
first  order  interactions  only.  Second,  the  authors  propose  a  technique  for 
validation  of  a  proposed  metamodel  whereby  a  simulation  run  is  deleted  and 
the  metamodels  recalculated  from  the  remaining  data.  They  state  that  "the 
significant  effects  should  remain  stable  upon  run  deletion,"  and  show  two 
ways  to  determine  the  stability  (Kleijnen  and  Standridge,  1988:260).  The 
first  technique  involves  a  qualitative  comparison  of  the  metamodels 
resulting  from  the  deletion  of  each  run;  while  the  second  compares  the 
predicted  response  of  the  model  obtained  using  all  runs  with  the  predicted 
response  of  the  model  obtained  when  one  less  run  is  used.  The  run  deletion 
technique  has  limited  applicability  to  the  current  research  because  128 
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design  points  are  involved  for  each  of  30  daily  metamodels,  for  both  the 

attack  and  non-attack  cases.  The  required  7680  recalculations  to 

investigate  the  stability  of  60  metamodels  is  prohibitive  in  both  computer  , 

and  researcher  time,  so  other  validation  techniques  must  be  used.  Third, 

the  authors  contrast  two  of  the  purposes  of  metamodeling:  prediction  and  • 

explanation  (Kleijnen  and  Standridge,  1988:260-261).  When  their  aim  was 

explanation,  the  authors  considered  validation  of  the  model  by  examining 

whether  the  coefficients  remained  stable  as  runs  were  deleted;  while  for  the 

prediction  criterion,  only  the  stability  of  the  predicted  value  of  the  response 

is  considered.  The  implication  is  that  when  prediction  is  emphasised,  there 

is  relatively  less  interest  in  which  coefficients  are  present  in  the  model  and 

the  values  they  take  as  long  as  their  combined  effects  result  in  good 

predictions. 

2.3.2  Europe  Container  Terminus.  Kleijnen,  van  den  Burg,  and  van 
der  Ham  (1979:50-64)  report  an  application  of  metamodeling  which  relates 
most  closely  to  this  research.  The  authors  note  that  at  the  time  of  their 
report  they  were  unaware  of  any  other  real-life  study  where  simulation, 
experimental  design,  regrest  *ox>  and  analysis  of  variance  techniques  had 
been  combined,  that  is,  their  study  was  one  of  the  first  practical 
applications  of  metamodeling  (Kleijnen  et  al.,  1979:63).  A  queuing  * 

simulation  was  used  to  model  the  required  container  stacking  capacity  at 
the  Europe  Container  Terminus  (ECT)  in  Rotterdam.  Metamodels  were  * 

then  derived  from  the  simulation  results  to  identify  the  factors  and/or  their 
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interactions  which  were  most  useful  in  explaining  the  behaviour  of  the 
simulation.  Several  similarities  exist  between  the  ECT  study  and  Diener's 

,  research.  Both  studies  involved  fractional  factorial  experimental  designs, 
although  the  ECT  study  only  required  six  variables  and  sixteen  runs,  and 

•  both  studies  drew  on  prior  research  to  identify  the  pool  of  potentially 
important  variables  (Diener,  1989:19-21;  Kleijnen  et  al.,  1979:57).  Another 
striking  similarity  between  the  studies  is  that  both  simulations  generated  a 
time  series  as  the  result  of  each  design  point.  In  Diener's  study,  the  time 
series  is  the  number  of  sorties  generated  on  day  one  through  day  thirty, 
while  for  the  ECT  study,  the  time  series  is  the  required  container  stacking 
capacity  measured  each  eight  hours  as  the  simulation  progresses  (Kleijnen 
et  al.,  1979:53).  The  total  duration  of  the  simulation  was  not  explicitly 
reported,  but  appears  to  have  been  one  year.  The  time  series  are  handled 
very  differently.  Kleijnen  et  al.  determine  the  frequency  at  which  a  given 
capacity  is  required,  and  calculate  the  mean  capacity  required,  as  well  as 
the  90th,  95th,  and  100th  percentiles  of  the  distribution.  Any  of  the  four 
measures  can  then  be  used  as  the  equivalent  of  a  single  dependent  variable 
to  characterise  the  time  series  generated  by  a  single  treatment  (1979:51,53). 
Such  an  approach  is  typical  in  describing  queuing  systems,  where  aggregate 

«  measures  are  used  to  describe  performance  rather  than,  for  example,  the 
queue  contents  at  a  series  of  specific  moments.  In  contrast,  Diener 

*  developed  a  set  of  thirty  metamodels,  each  of  which  represents  the 
important  effects  on  each  day  of  the  simulated  period,  and  is  calculated 
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using  the  simulated  responses  from  each  of  the  128  treatments.  The  two 
approaches  to  handling  time  series  data  represent  clear  opposites.  The  ECT 
study  implicitly  disregards  to  some  extent  the  importance  of  behaviour  over 
time  by  wrapping  up  each  time  series  into  one  variable,  while  Diener's  daily 
models  when  considered  together  clearly  show  the  varying  behaviour  of  the 
airbase  system  over  time  but  are  unable  to  represent  that  behaviour  over 
the  time  dimension,  as  each  model  is  a  separate  daily  snapshot. 


Starbird 


(1990:321-338)  reports  an  application  of  metamodeling  that  differs  greatly 
from  those  already  discussed.  The  simulation  model  of  a  tomato  processing 
plant  appears  relatively  simple,  with  only  three  independent  variables 
considered,  but  the  experimental  design  and  the  development  and  form  of 
the  metamodels  is  based  on  a  methodology  completely  different  from  the 
fractional  factorial  experiments  and  least  squares  linear  regression 
considered  so  far.  Starbird  used  the  Schruben-Cogliano  response  surface 
methodology,  which  "uses  frequency  domain  experiments  to  identify  the 
significance  of  particular  polynomial  terms  in  a  metamodel."  The 
metamodels  developed  were  the  generalised  Leontief  form,  which  Starbird 
states  is  a  "special  polynomial  form  often  used  for  the  modeling  of  cost 
relationships"  (Starbird,  1990:327).  In  a  frequency  domain  experiment  the 
input  factors  are  oscillated  at  carefully  chosen  frequencies  and  their 
importance  estimated  from  analysis  of  the  spectrum  of  the  response 
(Starbird,  1990:323).  Starbird's  work  shows  that  there  are  techniques  other 


t 
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than  those  based  on  regression  to  develop  metamodels,  but  it  is  dear  that 
the  techniques  described  by  Starbird  are  neither  applicable  to  the  current 
»  research  as  a  different  form  of  metamodel,  nor  would  they  be  applicable  as  a 

modification  of  Diener's  original  experiment.  Frequency  domain 
'  experiments  require  quantitative  variables  with  some  distribution  of  values 

over  which  they  can  be  oscillated,  and  the  flexible  metamodel  form  is  most 
applicable  to  cost  functions  with  relatively  few  variables  (Starbird, 
1990:327,328),  neither  of  which  criteria  are  met  by  the  ABO  problem. 

2.3.4  Multiple  Response  Experiments.  Friedman  suggests  that  "it  is 
a  rare  system  simulation  which  outputs  only  a  single  measure  of 
effectiveness  for  analysis,"  and  proposes  multivariate  statistical  techniques 
for  the  analysis  of  simulations  with  multiple  responses  (1983:1-2).  It  could 
be  considered  that  Diener's  experiment  falls  into  the  category  of  a  multiple 
response  experiment,  with  the  number  of  sorties  generated  on  each  day 
separate  measures  of  effectiveness  of  the  airbase  system,  but  there  is  at 
least  one  compelling  reason  why  this  condusion  should  not  be  drawn.  The 
reason  is  that  the  daily  sortie  rates  are  part  of  an  overall  time  series  with 
each  day's  result  the  same  measure  of  effectiveness  as  all  the  others,  but  at 
a  different  point  in  time.  From  this  perspective  a  multiple  response  might 
'  indude  additional  measures  such  as  total  hours  flown  per  day  and/or  enemy 

aircraft  destroyed.  These  hypothetical  measures  would  also  form  a  time 
series  if  the  results  were  reported  on  a  daily  basis,  so  that  multiple 
responses  would  only  exist  across  time  series,  not  within  them.  Although 
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Friedman's  multivariate  techniques  are  therefore  not  useful  for  this 

research,  it  is  worthwhile  to  note  that  in  developing  metamodels  to  relate 

the  multiple  responses  of  her  simulation  of  an  M/M/s  queuing  to  the  input  « 

factors,  Friedman  calculated  a  separate  metamodel  for  each  of  the  responses 

(1983:78-79;85).  Thus,  although  Friedman  was  able  to  test  the  models  * 

using  multivariate  techniques,  having  to  develop  three  separate  metamodels 

meant  that  "the  dynamic  interrelationships  among  the  response  variables 

were  not  used  directly  in  the  analysis"  (1983:85).  The  main  analogy 

between  Friedman's  research  and  Diener's  is  that  both  studies  resulted  in 

individual  metamodels  for  each  dependent  variable,  rather  than  a  single 

model  simultaneously  relating  all  the  responses  to  all  the  independent 

variables.  The  similar  result  from  the  two  studies  suggests  that  such 

simultaneous  analysis  of  more  than  one  response  with  all  the  input 

variables  is  a  particularly  difficult  task. 

2.3.5  Application  Summary.  Of  the  literature  reviewed,  the  Europe 
Container  Terminal  study  has  the  most  similarity  and  applicability  to 
Diener's.  The  only  real  similarity  in  the  other  research  is  that  simulations 
were  carried  out,  and  some  form  of  metamodel  developed.  The  features  of 
Diener's  research  that  distinguish  it  from  the  ECT  study  are  the  complexity 
in  number  of  variables,  the  qualitative  nature  of  all  the  variables,  and  the  * 

difficulty  in  finding  single  measures  to  characterise  a  time  series  response. 

The  magnitude  of  the  difference  in  complexity  can  be  appreciated  by  * 

considering  the  number  of  variables  and  their  possible  interactions.  For 
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models  with  three  variables,  only  three  interaction  terms  are  possible.  For 
the  six  variables  in  the  EOT  study,  fifteen  interactions  are  possible, 
although  prior  experimentation  had  reduced  the  pool  to  only  six  interactions 
so  that  a  total  of  twelve  coefficients  had  to  be  estimated.  For  the  ten 
variables  included  in  Diener's  study,  45  interactions  are  possible,  none  of 
which  could  be  discounted  a  priori,  resulting  in  a  total  of  55  coefficients  to 
be  estimated.  Reducing  the  large  number  of  candidate  terms  to  only  the 
important  terms  forms  a  large  part  of  the  rest  of  this  research. 

2.3.6  Metamodeling  Research  Issues.  There  is  some  recent  literature 
that  deals  with  important  research  issues  in  metamodeling.  Sargent 
(1991:889;892)  provides  lists  of  1)  properties  that  metamodels  exhibit, 
including  the  purpose  of  the  model,  whether  it  has  single  or  multiple 
responses,  whether  the  responses  are  deterministic  or  random,  and  how 
many  and  what  type  of  variables  are  considered;  and  2)  some  of  the 
decisions  that  must  be  made  in  developing  metamodels;  including  the  type 
of  metamodel  to  use,  what  criteria  to  use  for  evaluation  of  the  model, 
whether  screening  experiments  should  be  carried  out,  the  type  of 
experimental  design  to  be  used,  whether  the  metamodel  is  sufficiently 
accurate,  and  whether  the  metamodel  is  valid  with  respect  to  both  the 
simulation  model  and  the  real  system.  Several  of  the  decisions  that  Sargent 
lists  are  particularly  relevant  to  this  research,  while  others  have  already 
been  made  in  the  previous  research.  Decisions  to  be  made  in  this  research 
include  the  form  of  the  model,  and  whether  the  form  should  change  for 


different  purposes,  how  well  the  model  fits,  and  whether  it  is  sufficiently 

accurate.  Decisions  that  have  already  been  made  are  the  experimental 

design,  the  stochastic  response,  and  the  identification  and  type  of  possible  , 

variables.  Also,  this  research  is  unable  to  examine  the  validity  of  the 

models  against  the  real  system.  Sargent  also  provides  some  examples  to 

illustrate  some  of  the  issues  such  as  functional  form  and  goodness  of  fit. 

However,  he  only  considers  a  single  independent  variable  queuing  model, 
which  has  limited  relevance  to  the  multiple  regression  problem  in  this 
research  (Sargent,  1991:889-891). 

2.3.7-  Alternative  Regression  Model  Forms.  Barton  (1992;  1993) 
discusses  some  of  the  advantages  and  limitations  of  polynomial  regression 
models,  which  are  the  type  developed  by  Diener,  and  makes  some  useful 
observations.  First,  polynomial  models  are  relatively  easy  to  interpret  and 
therefore  "the  general  behavior  of  the  metamodel  is  easy  to  predict  from  the 
coefficients  of  the  polynomial"  (Barton,  1993:12).  Because  a  primary  aim  of 
developing  metamodels  for  the  ABO  problem  is  explanation  of  the  behaviour 
of  the  airbase  system,  ease  of  understanding  and  interpretation  should  not 
be  underestimated  in  the  choice  of  model  form.  As  a  disadvantage  of 
polynomials.  Barton  (1993:12)  states  that  "polynomial  metamodels  ...  are 
relatively  inflexible  for  fitting  general  non-linear  response  functions."  Such  *> 

a  disadvantage  will  only  become  a  consideration  in  this  research  if 
alternative  non-linear  forms  become  candidates  for  comparison  with  the  r 

existing  forms.  Barton  also  reviews  seven  different  types  of  models  as 
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alternatives  to  the  polynomial  form,  comprising  Taguchi  models,  generalised 
linear  models  (distinct  from  general  linear  models),  spline  methods,  radial 
t  basis  functions,  kernel  smoothing,  spatial  correlation  models,  and  frequency 

domain  approximations  (Barton,  1992:290).  Taguchi  models  require  a 
v  specialised  experimental  design,  and  so  are  excluded  from  further 

consideration.  Generalised  linear  models  allow  error  terms  to  come  from 
"any  exponential  family  other  than  the  Normal/Gaussian,"  and  may  be 
useful,  but  as  Diener's  existing  residual  analysis  supported  normality,  the 
additional  complexity  and  specialised  analysis  techniques  required  do  not 
justify  use  of  this  form  for  the  existing  data  (Barton,  1992:292).  Spline 
models,  radial  basis  functions,  kernel  smoothing  methods,  and  spatial 
correlation  models  require  independent  variables  that  are  continuous. 
Frequency  domain  models  were  previously  shown  to  be  inapplicable  to  the 
existing  data. 

2.4  Summary 

This  chapter  examined  some  of  the  key  issues  relevant  to  this 
research  of  Diener's  Air  Base  Operability  study,  including  Diener's  research 
objectives,  his  experimental  design,  and  the  form  and  assumptions  behind 
the  metamodels  he  developed.  Several  reported  applications  of  simulation 
metamodeling  were  reviewed,  as  well  as  some  research  issues  and  some 
possible  alternate  forms  for  metamodels.  The  application  literature  was 
found  to  have  few  direct  parallels  with  Diener's  study,  while  the  data 
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inherited  from  Diener  were  found  to  constrain  many  of  the  choices  that 
could  otherwise  be  made.  The  next  chapter  describes  the  available  data,  the 
analysis  of  the  existing  models,  and  the  investigation  into  alternate  models. 
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m.  Exploration 


*  3.1  Introduction 

This  research  is  a  largely  exploratory  study  into  1)  the  techniques 

*  relevant  to  the  development  of  regression  metamodels  from  the  existing 
data;  and  2)  the  development  of  alternative  metamodels  to  those  derived  by 
Diener  (1989),  with  the  aim  of  both  interpreting  the  behaviour  of  the  TSAR 
airbase  operability  simulation  model,  and  predicting  the  response  of  the 
model  to  a  set  of  input  conditions.  Because  of  the  wide  ranging  exploratory 
nature  of  this  research,  no  one  methodology  as  such  is  identified:  rather 
several  methodologies  are  used  and  evaluated,  and  the  applicability  of  each 
discussed.  The  exploration  is  divided  into  two  phases. 

In  the  first  phase  the  existing  data  are  presented  and  limitations  on 
the  research  caused  by  the  form  of  the  data  and  the  experimental  design 
from  which  it  derives  are  discussed.  Further  analysis  of  the  original 
metamodels  developed  by  Diener  (1989)  is  carried  out. 

The  second  phase  examines  techniques  for  developing  new 
metamodels,  including  backward  elimination,  forward  selection,  and 
stepwise  selection.  All  possible  regressions  cannot  be  calculated  due  to  the 

*  large  number  of  possible  variables  and  interaction  terms;  however,  best 
subsets  methods  are  considered,  using  Mallow's  Cp  and  adjusted  R2  as  model 

*  selection  criteria.  A  preliminary  evaluation  of  model  building  techniques 
and  selection  criteria  is  carried  out,  leading  to  a  revision  of  the  techniques 
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initially  proposed.  Finally,  new  models  are  developed  and  evaluated  using 
both  explanation  and  prediction  as  assessment  criteria. 

3.2  The  Existing  Database 

As  stated  in  Chapter  1,  the  existing  database  comprising  Diener's 
experimental  design  and  the  results  of  his  simulations  forms  the  basis  for 
this  research.  Awareness  of  how  the  data  was  produced  and  how  it  is 
structured  is  a  prerequisite  to  understanding  the  existing  models  and 
developing  subsequent  models. 

3.2.1  Data  Generation.  The  21y3  fractional  factorial  experimental 
design  discussed  in  Chapter  II  contains  128  design  points,  each  of  which 
represents  a  unique  combination  of  the  ten  input  variables  and  defines  the 
initial  settings  for  a  run  of  the  simulation  model  over  a  thirty-day  period. 
The  simulation  records  the  number  of  sorties  flown  on  each  day  of  the 
thirty-day  period,  so  that  each  of  the  128  design  points,  or  treatments, 
produces  thirty  data  points.  All  128  treatments  are  applied  twice,  once  with 
attacks  on  the  base  modeled  by  the  simulation  and  once  without  attacks, 
and  the  data  from  the  attack  and  no-attack  cases  are  analysed  separately  as 
two  sub-experiments  (Diener,  1989:46). 

3.2.2  Data  Structure.  The  data  may  be  arranged  in  several  ways, 
but  the  most  compact  form  for  each  case  is  a  matrix  of  128  rows  and  47 
columns.  The  first  30  columns  contain  the  response  variable,  sorties  flown, 
for  day  one  through  thirty;  and  the  last  seventeen  contain  the  design 
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matrix,  with  ten  columns  for  the  independent  variables,  and  seven  columns 
for  variables  representing  the  blocking  effects.  The  128  rows  in  the  matrix 
represent  each  of  the  treatments  required  by  the  design.  Figure  3.1  lists 
the  variables  that  appear  in  the  design  matrix,  while  Figure  3.2  graphically 
shows  the  data  structure.  Diener  (1989:27-42)  provides  detailed  information 
about  the  factors  and  their  levels.  Interaction  tenns  are  developed 
automatically  during  the  regression  process  by  creating  45  new  variables, 
the  values  of  which  are  the  product  of  the  appropriate  main  effect  values. 


|  Factor 

A 

Level  of  attrition  experienced  1 

B 

Availability  of  filler  (replacement)  aircraft 

C 

Aircraft  battle  damage  repair  (ABDR)  capability 

D 

Recovery  capability  from  air  base  attack 

E 

Maintenance  personnel  numbers 

F 

Avionics  Intermediate  Test  Stations  (AIS) 

G 

Support  equipment 

H 

Spare  parts  stocking  levels 

J 

Missile  stocks,  components  and  deliveries 

K 

Fuel  initial  stocks  and  resupply  schedule 

Figure  3.1  Design  Matrix  Factors 
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Sorties 

Variables 

Block  Terms  | 

Treatment 

Day  1  Day  30 

abcdefghjk 

B1  B7  | 

1 

226  45 

1  0  0  0  0  0  0  1 

] 

128 

187  68 

hmbhhh 

Figure  3.2  Data  Structure  Summary 


Two  points  are  apparent  from  the  structure  of  the  data.  First, 
looking  across  the  matrix,  each  of  the  128  treatments  results  in  a  30  day 
time  series  of  sorties  flown;  and  second,  looking  down  the  columns,  for  each 
day  128  responses  are  available  to  estimate  the  effects  of  the  factors  applied 
differently  in  each  treatment.  Diener's  analysis  took  the  second  viewpoint 
in  estimating  important  factors  and  their  effects  in  thirty  daily  metamodels 
(Diener,  1989:50).  This  research  will  initially  take  the  same  viewpoint  as 
Diener.  To  take  the  first  viewpoint  and  estimate  the  factor  effects  across 
time  is  a  significantly  different  problem,  and  is  beyond  the  scope  of  this 
research. 

3.3  Model  Form  and  Experimental  Design 

3.3.1  Linear  and  Non-Linear  Models.  As  a  term  in  fairly  common 
use,  linear  regression  can  be  somewhat  misleading.  The  word  linear  applies 
strictly  to  the  parameters,  or  coefficients  of  the  model.  For  example,  both 
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the  following  models  are  linear  regression  models  because  the  parameters  (J 
are  neither  multiplied  together,  nor  raised  to  a  power  other  than  one. 

Yt  =  %  +  ft*  +  M*  +  ei  (3.1) 

¥>  =  %  +  Mi  +  M*  +  PX  +  PX  +  PAX  +  e,  (3.2) 
Even  though  the  second  model,  (3.2),  contains  quadratic  and  interaction 
terms  for  the  independent  variables,  while  the  parameters  remain  linear, 
and  an  additive  error  term  is  included,  the  model  meets  linearity 
requirements.  The  exponential  model  below  is  non-linear,  because  the 
parameter  y2  appears  as  an  exponent. 

Yi  =  y0  +  ytexp(y^Q  +  e*  (Neter  et  al.,  1985:468).  (3.3) 

The  exponential  form  of  model  is  unsuitable  for  the  data  from  Diener's 
research  because  it  provides  for  only  one  independent  variable. 

3.3.2  Experimental  Design  Limitations  on  Model  Form.  The 
experimental  design  used  by  Diener,  previously  discussed  in  Chapter  n, 
imposes  some  limits  on  the  form  of  metamodels  that  can  be  developed  from 
the  experimental  results.  Curvature  (quadratic  and  higher)  terms  are 
precluded  because  the  experimental  design  makes  the  de  facto  assumption 
that  there  is  a  linear  relationship  between  each  independent  variable  and 
the  model  response.  The  linearity  assumption  is  a  result  of  the 
experimental  design  allowing  only  two  levels  for  each  independent  variable, 
so  that  a  linear  relationship  is  all  that  can  be  deduced  from  the  data.  As 
Kleijnen  points  out,  "a  second-order  model  with  pure  quadratic  effects 
requires  three  or  more  levels"  (Kleijnen,  1987:334).  The  relationship  can  be 
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visualised  by  considering  the  extreme  case  of  just  one  data  point  at  each 

level.  It  is  dear  that  the  two  responses  can  only  form  the  ends  of  a  straight 

line,  as  there  are  no  intermediate  points  to  indicate  curvature.  The 

regression  equation  also  shows  that  curvature  information  cannot  be 

obtained  from  the  model.  Consider  a  variable  x^  which  takes  the  levels  -1  , 

and  1  in  the  design  matrix.  If  we  try  to  indude  a  quadratic  term  x^  in  the 

model,  its  value  is  always  1,  the  same  as  the  high  level  of  the  linear  term 

xlt  and  therefore  no  additional  information  is  provided. 

3.3.3  Experimental  Design  and  Interactions.  Higher  than  two-way 
interactions  cannot  be  induded  in  the  metamodels  because,  in  the  chosen 
fractional  factorial  design,  the  higher  interactions  are  confounded  with  each 
other,  as  discussed  in  Chapter  II.  The  previous  researcher  did  not  consider 
the  inability  to  indude  higher  order  interactions  to  be  a  serious  restriction, 
and  points  out  that  limiting  the  analysis  to  main  effects  and  two-way 
interactions  is  common  practice,  as  the  interpretation  of  higher  order 
interactions  is  very  difficult  (Diener,  1989:44;  1993). 

3.4  The  RTiflting  MfttamndAla 

3.4.1  Coding  of  the  Design  Matrix.  In  developing  the  existing 
metamodels,  Diener  originally  coded  the  design  matrix  so  that  the  low  level  » 

of  a  factor  was  represented  by  a  0  and  the  high  level  by  a  1.  For  example,  if 
the  first  treatment  required  factors  A,  C,  and  E  at  their  low  level,  with  all  » 

other  factors  high,  the  first  row  of  the  design  matrix  was  coded  as 
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0101011111.  Using  (>1,1)  coding,  the  first  treatment  in  the  example 
above  is  now  coded  -1  1  -1  1  -1  1  1  1  1  1.  To  determine  whether  the  use  of 
♦  (-1,1)  coding  has  a  significant  effect  on  the  resulting  models,  metamodels 

are  calculated  using  the  same  technique  used  by  Diener,  i.e.  backward 
r  elimination,  with  variables  retained  in  the  model  at  a  significance  level  of 

0.10,  but  with  the  design  matrix  modified  for  (-1,1)  coding.  The  resulting 
models  are  compared  with  the  existing  models  for  significant  variables 
included,  goodness  of  fit,  and  estimation  of  sorties  generated.  Figures  3.3 
and  3.4  show,  for  the  attack  and  no  attack  cases,  the  variables  included  in 
the  metamodels  in  increasing  order  of  partial  R2,  which  is  the  marginal 
contribution  to  the  explanation  of  variance  by  a  variable,  given  that  all 
others  are  included  in  the  model  (Neter  et  al.,  1989:285).  Table  3.1  lists  the 
adjusted  R2  for  each  model,  and  Table  3.2  and  3.3  compare  the  predicted 
values  of  sorties  generated  for  all  factors  at  their  low  level  and  all  factors  at 
their  high  level. 

3.4.2  Significant  Variables  Included.  Attack  Case.  For  the  attack 
case,  except  for  the  first  day,  the  two  coding  methods  never  result  in 
identical  models,  and  none  of  the  models  is  a  subset  of  its  counterpart.  The 
most  striking  difference  is  that  although  the  (0,1)  models  generally  contain 
more  terms,  the  terms  most  frequently  omitted  in  the  (0,1)  models  are  main 
effects  that  are  consistently  the  most  important  in  the  (-1,1)  models.  For 
example,  variable  B  (filler  aircraft)  is  the  most  significant  factor  in  24  of  the 
30  (-1,1)  models,  yet  does  not  appear  in  eight  of  the  (0,1)  models  for  the 
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appropriate  days.  Variable  G  (support  equipment),  which  is  the  second 
most  important  variable  on  twenty  days  when  coded  (-1,1)  is  also  omitted 
eight  times,  while  variable  H  (spares  stocking  levels),  which  is  at  least  the 
fourth  most  important  on  fifteen  days  when  coded  (-1,1),  is  omitted  on 
eleven  of  those  days,  and  on  four  other  occasions.  Particularly  in  the  last 
twelve  days,  the  (0,1)  models  appear  to  capture  the  effect  of  variables  B  and 
H,  by  including  their  interaction  term  BH  as  the  most  important  effect. 

3.4.3  Significant  Variables  Included.  No-Attack  Case.  As  for  the 
attack  case,  no-attack  case  models  derived  from  the  (0,1)  design  matrix 
generally  contain  more  terms  than  their  counterparts.  The  tendency  to 
ignore  a  main  term  significant  in  the  (-1,1)  models,  and  include  an 
interaction  term  containing  that  variable  is  also  apparent.  For  example, 

BH  is  often  included  as  an  important  term  in  the  (0,1)  models  on  days  when 
the  (-1,1)  models  include  both  B  and  H.  Finally,  the  (-1,1)  models  generally 
contain  fewer  interaction  effects  that  are  present  without  at  least  one  of  the 
relevant  main  effects,  and  on  average  contain  more  main  effects  relative  to 
interaction  effects. 

3.4.4  Comparison  of  Adjusted  Ra.  It  is  apparent  from  Table  3.1  that 
the  models  derived  from  the  (0,1)  design  matrix  generally  achieve  slightly 
higher  values  of  adjusted  R3,  with  the  greatest  differences  usually  occurring 
when  the  (0,1)  model  has  at  least  three  terms  more  included  than  the  (-1,1) 
model.  As  the  maximum  difference  in  R3  for  all  days  across  both  cases  is 
only  0.05,  with  an  average  difference  of  0.014,  both  sets  of  models  can  be 
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L 

Variables  Included 
with  (0,1)  Coding 

Variables  Included  1 

with  (-1,1)  Coding  | 

d 

d 

dj,cj,c,b,bh,hj,ce,h,be,ek 

d,q,bh,cehj»ag.be 

djb,b,bh,be,c^,g 

d,bh,qj,e 

djbh&cf,egbkjk,e,kj 

d,bh,eg,hk)gjkiaf,c,cf 

bh,d,hk,g,ce,e>bc>bj,kj 

b,d,bh^,bf,ehkhj,ce 

6 

gJ*Aj,bkj^g,ejk^ 

g,bhj,bh,e,hkjk,fl,fg,bk 

7 

bg^k,<g,eM4grhk,ef 

b,g,hk,hj,ahh,be 

8 

b*k,a,ce,e,d},cj,ak 

b,ak,gIce,e,qIef 

9 

b,g,eh,ae,ak,ag,e,a,k 

b,g,e,ah,ae,ak,ag 

10 

bg,bjM,e,k,g,ab,hk 

b^hk.bj.e^ 

11 

b^,q4k,a,dejk,k,hk^ah,dg^i^k 

b,g,akjk,cj,ef,hk,c,ktfk 

12 

b,g,fhjk,ef,ak,bd,fk,q,e,kfhk,a 

b,g,h,ak,hk,fk,bh,efjk 

13 

b,g,bh,a,ej?kjk,ak 

b,gjk,ej,ak,bh 

14 

b,g,bhjk,ej,cd,fk,cgIef,dh,ah,c 

b^h4k,ah,ej,cg 

16 

b,gk)bh,ak,ej,ajk,fk)crac 

b,g,ak,ej,h,cjk,fk,ac 

16 

b,g,bh,fk,q,jk,ej,ef 

b,g,h^k,ak,ejjk,hk,c,bh 

17 

bh^,bk,ak,cj,fg,ejA,bjjk,hk,gk>eg 

b,g,akjk,fk,h/g,eg,bk,hk 

18 

b,gj,fh,ef,gk,ejjk,cd,bh4k>dh,hk,c 

b,g,h,gk,ej,hk,cdjkj 

19 

bh,b,g,ej,c(ty,fk,gkTfg1hkjk,c8 

b,gjk,h,bh,ejjfkj 

EM 

bh^,bk,eg^kjk4Vg,ej4k^ak^c,ce 

b^jk4k,ej,bk,ak,bh^kh,fb,ce/g 

*» 

bh4k^k.b^h,ej,cgjk,eg,fg^j^k 

b,g,h^'k,ej,fktakjk,be,bh,ctfh,ce 

bh^,ej,b^5^h,cg^g,eg,be^kjk,bk^g^k, 

fk 

b,g,h,ej,ak,fk,gkjk,fh,c,hk,eg,bh 

■ 

bh4jk,b,cg,hk,fli,be,fk,fg,ejjk 

b.g.hjkej.fe.hk.ak.fk 

24 

bh&fh,bk,hkh,fg,fk,ck,gk,eg,ce 

bIgh,bh^,ej,fk,bk,ce,eg,f,fh,dg 

25 

bh^k,ce^i,ejjk4kb,ag,b,c,ak/g 

b,fk,h,ej,g,bh,gk,hkjk 

26 

bh£jhk,jk,gk,ce,eg,fk,ej,c 

b,g,bh,h,ej,fk,gkjk,ce,hk,eg 

27 

bh,g,b,ak,ce,fkjXegIej,gk,eh,c,dhthk,bk, 

b^h,ej,fk,bh,ce^kjkhk,f,c,bk,eg 

28 

bh,c,bAgjk^k^,ce,ej^hk,bg/g 

b/kh,ej^jk^k,cehkrf,bg 

29 

bh,c,hk,gkjk,ej,bk,fk,ak,dh,ce,g,fg,dg 

b4kh,ej^jk^k,ce,hk/,bg,fh,fg 

30 

bh^,b^k^gjk,ej,be,bg,c 

b,fk,h,ej,g,fg,bhjk,be,bg,ce,hk 

(Diener,  1989:307-336) 


Figure  3.3  Comparison  of  Variables,  Attack  Case  Reduced  Models 
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r 

Day 

Variables  Included 
with  (0,1)  Coding 

Variables  Included 
with  (-1,1)  Coding 

a,q,cf,g/,ag^bd,cd,bf,fg 

ac4gj,ag,cd,tf,a 

e,ak,j,g,eh,b,bd,d^jIbj,dj 

e,dj^],eh,bj,bdjk^ 

e.hk.bj.ak.dj.df.k.dh.a.bk.bf.bh 

e,hk,ak,dh,dj,dAbk,bf,eg,ab,bj 

e,bjjkj,ek,df,q  ,eh,bh 

e.bjkj.c.q 

5 

e,bjbh,fai,h,ck,ef,bk,ad,fh,ah 

e,b,bh,cgXbj.ej,ef 

6 

be,bj,ae,ef,ab,k,ad,tf,ah,hk,fk,bc,q,c,f,bg, 

eg,bd,ek,ak 

e,bjbj,q,h,fk>j,ak,a,c,ad,hk,ef,be,ab 

7 

bj,ek,bd,bcj,dk,ej,ce,bg^g,g 

b,e,q,ce,ag 

8 

bj,be,ej,bg,gh,df,eh,hj,a,ag,d 

j.bj,ej,gh,eh,b,bg,f,ag 

9 

bj,e,b,ej,eg,hk,de,c,cf,ae,ak,h,fg 

j,ej,bj,e,eg^k,dj 

10 

b,ejji,bjf,ef,aggbjg,egtfkfab,ek 

bj,ej^bj/^g,eg,ef^i,e,ac4g 

11 

b,hj,h,ek,k,q,a,ck,ab 

b,hj,e,hj,k,ab 

12 

b,dh,eg,dg,af^hj^gj 

b^gj,e^i,eg,ae,dg,af,ac,dh 

13 

b,ch,fh,ce,ac,ae,dk,gh,gk,hj 

b,h,e,dk,ce,fh,£u,cd,acj 

14 

b^,be,bd,dh,cj 

b,h,be,e,bd,dh 

15 

b,h,ab,eh,bj>bf,fk,cd,ae,bd 

b,h,ae,a,ab,cd,k,bj,eh,bf 

16 

b,bhJrc,be,ce,ej,aj,ab,g,fh^,dk,c,(Ug 

b,h,bc,aj,dk,ce,ab,bh,be,g,ek 

17 

b^h^h.bj.dh.d.W.bgJk^.cg.ce^kJ^.dj 

b,h,]q,fh,bf,ce,hj,dh,dj,cg,g 

18 

b,bf,e,cf,gh4y,be,ef,bc,g,eh^ui,fk,f,bg,bd, 

akjj 

b,be,be,cf/Xbf^j4k4y>f,bg^eIck^h,eh 

19 

bk,€U,bc,dh,eh,ck,k,ek.hk,dk, eg.bj.de, ce,hj, 

b,ej,bk,hkjk,ej,bc,k,ce,dk,gj,ek,c,ck,eh, 

gjjkjb 

h 

FI 

bk, bd.ej.de, ce,dIc,bj,dg,fk,gh,acjk,bg,b 

b,k,bk,cej,dg,bd,djk,de,ej,e,f,ek,bj  .fj.bg 

r 

b4>h,bk,fh,e,(j,rf,gh,bdjk,ae,te,ly,gj,hk,fg, 

bc,eh,f,dj,ad 

b,h,be,f3,bk,fh,bh4^hk,eh,cf,hj,ghjk,ae 

r 

b,bk,h,bg,hjj,bc,ak,eh4ik,c,e 

b,h4y,k,bg,bk,ak^g,bc,eg,eh 

b,bh,ek,gj,c^,de,bc/g4j,hj,egjk,dh£ac 

b,h,e,bh,ac,hj,ek,k,fg,fj,bc,eg 

24 

b,ek,ab,f,bh,eh,cd,q,ej,e,ag,ah45,bd 

b,e,§,k,ag,q,aj,ah,ab,h,ek,ac 

25 

b,be,bj,ck,ac,a,ch,e,dh,ae 

b,e,ac,hj,ae,kj,be,dh 

26 

b, be, bh,bc,bj,ef^k,dj/g,q,cdjc, hk.dk  ^h^j 

b,e,be,bh,c,bj,bcjb,qj,dk 

27 

be,bk,bj,bh,ef,d,hj,c,gj,ce,de 

bk,e,bh£,dej,bj,c,ag,q,h,ly,f,gh,ce 

28 

b.bk.eh.ekjT^bh.qj.af.df.ck.bc.ly.ag 

bk.k.bjk.bh.e.hj.af.q.h.ag.df.bc 

29 

k.bhjk.be.ek.fk.dk.hj.ag.ad.d.q.bj.ck.f 

bk,k,b,jk,bh,ag,e,fk,dk,hj,h,j,c 

m 

jk,b,bh,bk,ek,gj,t!j,ly,hj 

k,bk,bjk4i4ilj.bj,bhj,e,ek 

Figure  3.4  Comparison  of  Variables,  No-Attack  Case  Reduced  Models 
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considered  as  equivalent  in  explaining  variance.  Adjusted  R2  was  chosen  as 
the  criterion  for  comparison  because  it  considers  the  loss  in  degrees  of 
«  freedom  as  additional  variables  are  added  to  a  model,  thus  increasing  R2, 

but  possibly  not  adding  any  useful  information.  The  usefulness  of  the 
'  adjusted  R2  value  is  somewhat  reduced;  however,  because  with  so  many 

terms  (55  including  two-way  interactions)  available  for  inclusion,  the 
addition  of  extra  terms  with  only  marginal  contributions  to  R2  results  in 
little  adjusted  R2  penalty  because  the  relative  change  in  degrees  of  freedom 
is  small. 

3.4.5  Estimated  Values  of  Sorties  Generated.  For  the  attack  and  no¬ 
attack  cases  respectively.  Tables  3.2  and  3.3  show  for  five  arbitrarily 
selected  days  that  the  number  of  sorties  estimated  is  not  consistent  between 
the  two  schemes.  The  difference  in  estimated  sorties  is  not  unexpected 
given  the  differences  between  the  models  in  variables  included.  The  effect 
of  the  blocking  terms  has  been  disregarded  because  their  effect  is  constant 
across  both  cases  and  both  coding  schemes. 

3.4.6  Variance  nf  Estimated  Sorties  Generated.  Depending  upon  the 
coding  scheme  used,  there  are  important  differences  in  the  calculation  of  the 
variance  of  the  estimated  value.  If  the  covariance  matrix  of  the  parameter 
estimates  is  calculated  for  the  (-1,1)  coding  scheme,  all  covariances  are 
found  to  be  zero  except  between  the  blocking  terms.  Also,  all  estimators 
including  the  intercept  have  equal  variance,  and  all  seven  blocking  terms 
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Table  3.1  Adjusted  R2  Values  for  (0,1)  and  (-1,1)  Design  Matrices 


Day 

Attack 

(0,1) 

Attack 

(-1,1) 

No-Attack 

(0,1) 

No-Attack  | 
(-1,1) 

0.69 

0.69 

0.49 

0.47 

0.92 

0.92 

0.52 

0.52 

0.76 

0.76 

0.54 

0.54 

0.56 

0.56 

0.42 

0.39 

5 

0.56 

0.56 

0.46 

0.44 

6 

0.39 

0.40 

0.67 

0.63 

7 

0.48 

0.47 

0.42 

0.39 

8 

0.40 

0.40 

0.43 

0.42 

9 

0.47 

0.47 

0.69 

0.67 

10 

0.46 

0.46 

0.73 

0.73  1 

11 

0.56 

0.53 

0.71 

0.70 

12 

0.60 

0.58 

0.70 

0.70 

13 

0.57 

0.57 

0.75 

0.74 

14 

0.57 

0.54 

0.65 

0.64 

15 

0.56 

0.55 

0.71 

0.70 

16 

0.55 

0.57 

0.75 

0.74 

17 

0.57 

0.54 

0.78 

0.76 

18 

0.55 

0.51 

0.77 

0.77 

19 

0.54 

0.50 

0.60 

0.58 

IBS® 

0.60 

0.56 

0.64 

0.63 

21 

0.63 

0.59 

0.75 

0.72 

22 

0.60 

0.57 

0.67 

0.68 

23 

0.52 

0.50 

0.73 

0.71  1 

24 

0.60 

0.58 

0.68 

0.68  9 

25 

0.60 

0.57 

0.58 

0.57  1 

26 

0.57 

0.56 

0.55 

0.50  | 

27 

0.66 

0.64 

0.40 

0.39  | 

28 

0.59 

0.58 

0.56 

0.53  1 

29 

0.57 

0.55 

0.64 

0.61  I 

30 

0.57 

0.57 

0.62 

0.61  | 
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Table  3.2  Comparison  of  Estimated  Sorties,  Attack  Case 


(0,1)  Design 

(-1,1)  Design 

(0,1)  Design 

(-1,1)  Design  jj 

All  Factors 

All  Factors 

All  Factors 

All  Factors  | 

Day 

Low 

Low 

High 

High  | 

1 

78 

78 

102 

102 

5 

72 

88 

131 

116 

10 

108 

109 

175 

156 

59 

48 

116 

110 

30 

23 

32 

58 

54 

Table  3.3  Comparison  of  Estimated  Sorties,  No-Attack  Case 


Day 

(0,1)  Design 

All  Factors 

Low 

(-1,1)  Design 

All  Factors 

Low 

(0,1)  Design 

All  Factors 

High 

(-1,1)  Design 

All  Factors 

High 

1 

264 

268 

259 

260  1! 

5 

177 

181 

216 

215  1 

10 

140 

131 

199 

205  1 

20 

98 

57 

131 

131  1 

30 

63 

28 

58 

63  1 

have  equal  variance.  The  variance  of  the  estimated  value,  using  a  two 
variable  example  for  simplicity,  is  calculated  as  shown  by  equation  3.4. 
s?(Yhat)  =  ^(bo)  +  Xfsffb,)  + 

+  2XIs(botb1)  +  2X2s(b0>b3)  +  2XIXas(b1,ba)  (3.4) 

where  s2(Yhat)  is  the  variance  of  the  estimate  for  sorties  generatdd^s2(b0)  is 
the  variance  of  the  intercept  parameter  estimate;  s2^)  and  s2(ba)  are  the 
variances  of  the  other  parameter  estimates  (including  one  blocking  term); 
and  s(b0,bj),  s(b0,b2),  and  s(bt,b2)  are  the  covariances  between  the  estimates 
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(Neter  et  al.,  1989:259).  Extending  the  example  to  consider  the  multiple 
variables  in  this  study,  we  see  that  with  all  covariances  either  zero  or 
disregarded,  all  X  values  either  -1  or  1  (so  that  all  X2  are  also  1),  and  all 
estimator  variances  equal: 

s?(Yhat)  =  s?(B)  +  (p+l)s?(b)  (3.5) 

where  p  is  the  number  of  predictors  in  the  model  (not  including  the 
intercept);  s2(b)  is  their  common  variance;  and  s^B)  is  the  variance  of  the 
blocking  term.  Note  that  the  variance  of  the  intercept  term  and  the 
variances  of  the  predictors  are  all  equal.  Because  a  given  combination  of 
inputs  can  only  come  from  one  block,  only  one  variance  term  need  be 
included,  and  the  covariance  between  the  blocking  terms  can  be 
disregarded.  There  is  no  covariance  between  the  blocking  terms  and  the 
predictors.  The  value  of  s2(Yhat)  does  not  change  regardless  of  the 
combination  of  input  levels  chosen. 

In  contrast,  models  developed  using  (0,1)  coding  show  covariance 
between  all  variables,  greatly  complicating  the  calculation  of  variance  for 
the  estimated  value  of  sorties  generated,  except  when  all  variables  are  at 
their  low  level,  i.e.  zero,  when  reference  to  equation  3.1  shows  that  the 
variance  of  the  estimated  response  collapses  to  the  variance  of  the  intercept 
estimate  plus  the  variance  of  a  single  blocking  term.  Thus  s2(Yhat)  is 
different  depending  upon  the  particular  set  of  factor  levels  in  which  we  are 
interested.  An  exhaustive  comparison  of  actual  variances  was  not  carried 
out,  but  examination  of  the  parameter  estimate  standard  errors  for  all 
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» 


models  shows  that  the  (-1,1)  model  standard  errors  are  approximately  half 
those  of  the  (0,1)  models,  although  the  (0,1)  models  may  have  lower 
variance  totals  for  the  all  factors  low  situation  because  only  the  intercept 
and  <me  blocking  term  are  considered.  At  other  design  points  it  is  likely 
that  the  (-1,1)  models  will  have  lower  overall  variance. 


For  simplicity,  the  preceding 


discussion  of  variance  has  considered  estimation  of  sorties  generated.  More 


rigorously,  the  estimation  is  a  point  estimate  of  the  expected  value,  i.e.  the 
mean,  of  the  distribution  of  sorties  flown  under  a  chosen  set  of  initial 
conditions.  For  the  linear  regression  models  used,  we  assume  that  the 
distribution  is  normal.  Calculation  of  the  variance  of  the  estimated  value 


allows  confidence  intervals  to  be  formed  for  the  point  estimate  of  the 
expected  value  of  sorties  flown.  When  predicting  a  new  value  for  sorties 
flown,  however,  we  must  consider  that  we  are  predicting  a  single  value 
drawn  from  a  distribution,  the  mean  of  which  we  are  already  estimating  by 


a  confidence  interval.  The  variance  of  a  predicted  value  is  therefore  greater 
than  the  variance  of  the  expected  value,  and  is  calculated  as  follows: 


s*(Y „J  =  MSE  +  g*(Yhat)  (3.6) 

where  Yw  is  the  new  prediction  of  sorties  flown,  and  MSE  is  the  mean 
square  error  of  sorties  flown  (Neter  et  al.,  1989:79-83).  Examination  of  the 
regression  results  shows  that  MSE  is  the  dominant  term  in  equation  3.6, 
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and  that  there  are  only  small  differences  between  MSE  for  equivalent  (0,1) 
and  (-1,1)  models. 

3-fi  Phase  Qna  Summary 

We  have  seen  that  the  nature  of  the  available  data  limits  our  analysis 
to  polynomial  least  squares  regression  models.  Comparison  of  the  two  sets 
of  models  highlights  the  issue  of  prediction  versus  explanation  discussed  in 
Chapter  I.  From  the  point  of  view  of  trying  to  explain  the  behaviour  of  the 
simulation,  the  two  coding  schemes  produce  different  results.  The  greater 
number  of  variables  included  in  the  (0,1)  models,  and  the  frequent  inclusion 
of  an  interaction  term  at  the  expense  of  either  or  both  of  the  relevant  main 
effects  makes  the  (0,1)  models  more  difficult  to  interpret,  and  in  some  cases 
leads  to  different  conclusions  about  important  effects.  From  a  prediction 
point  of  view,  however,  the  two  schemes  produce  broadly  equivalent  results, 
at  least  for  the  two  extreme  design  points  evaluated.  Comparison  of  the  two 
coding  schemes  has  shown  that  the  (-1,1)  design  matrix  is,  as  suggested  in 
Chapter  n,  more  appropriate  because  it  results  in  models  that  are  easier  to 
understand  and  interpret,  and  have  more  easily  calculable  estimation  and 
prediction  variances.  Therefore,  the  models  developed  using  the  (-1,1)  coded 
design  matrix  and  backward  elimination  at  a  significance  level  of  0.10 
become  the  baseline  for  development  of  new  models  in  the  second  phase  of 
this  exploration. 
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In  this  section,  some  of  the  considerations  that  are  important  in  the 
development  of  regression  models  are  discussed.  Initially,  the  assumption 
that  interaction  terms  are  important  is  examined,  because  the  task  is 
greatly  simplified  if  interaction  terms  can  be  discounted.  The  process  of 
reducing  the  variables  in  a  model  from  all  potential  predictors  to  a  subset 
that  best  achieves  the  aim  of  the  model  is  discussed,  including  the  rationale 
for  variable  reduction,  the  impact  of  variable  reduction  on  variance  and 
bias,  several  model  reduction  techniques,  and  variable  selection  criteria. 


underlying  assumption  of  Diener's  research  is  that  interactions  between  the 
factors  that  affect  ABO  are  very  important  in  the  analysis  of  the  sortie 
generation  problem  from  a  system  viewpoint.  A  relatively  simple  approach 
to  testing  this  assumption  is  to  calculate  daily  metamodels  with  all  main 
effects  and  interactions  forced  into  the  model,  and  then  to  test  the 


significance  of  this  full  model  against  a  reduced  model  containing  only  main 
effects  using  the  general  linear  test  (GLT).  The  null  hypothesis  Hq  is  that 
all  interaction  term  parameters  equal  zero,  or,  referring  to  equations  2.1 
and  2.2,  that  on  day  i,  all  45  ^(i)  for  the  no-attack  case  and  all  45  P*k(i)  for 
the  attack  case  equal  zero.  The  test  statistic  F*  generated  by  the  GLT  is 
calculated  as  follows: 


SSEm-  SSEr 


(3.7) 
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where  SSEr  is  the  error  sum  of  squares  for  the  fitted  model  using  only 

blocking  and  main  effects  (the  reduced  model);  SSEF  is  the  error  sum  of 

squares  for  the  fitted  model  using  all  effects  including  interactions  (the  full 

model),  and  dfR  and  dfF  are  the  degrees  of  freedom  for  the  respective  error 

sums  of  squares.  The  decision  rule  for  the  GLT  is  to  not  reject  Hq  at  „ 

significance  level  a  if  F*  £  F(1  -  a;  dfR  -  dfF,  dfF)  (Neter  et  al.,  1988:98-99). 

Tables  3.4  and  3.5  provide  the  results  of  the  tests  for  the  first  six 
days.  On  five  out  of  six  days,  for  both  the  attack  and  no-attack  case,  the 
test  fails  to  reject  Hg.  Given  our  prior  knowledge  of  the  existing 
metamodels,  many  of  which  include  interaction  terms  among  the  most 
significant,  failure  to  reject  Hq  is  surprising  until  we  consider  the  relative 
numbers  of  main  and  interaction  effects.  All  45  interaction  effects  are 
tested  against  just  ten  main  effects,  a  few  of  which  regularly  contribute  the 
most  to  the  explanation  of  variance.  The  GLT  assesses  the  significance  of  a 
group  of  predictors  by  the  increase  in  error  sum  of  squares  (SSE)  relative  to 
the  increase  in  degrees  of  freedom  when  the  predictors  are  removed  from 
the  full  model.  The  very  large  increase  in  degrees  of  freedom  of  the  reduced 
model  is  likely  to  mask  the  effect  of  a  few  important  terms  among  the  many 
which  are  insignificant,  and  unless  many  interaction  terms  are  significant, 
or  a  few  are  very  significant,  we  can  expect  not  to  reject  Ho.  Tests  for 
subsequent  days  are  not  carried  out  because  the  example  of  the  first  six 
days  adequately  illustrates  the  disproportionate  effect  on  degrees  of  freedom  • 
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of  the  very  large  pool  of  interaction  terms  when  compared  to  a  much 
smaller  number  of  main  terms. 


Table  3.4  General  Linear  Test  for  Interaction  Terms,  Attack  Case 


Reduced 

SSE 

Full 

SSE 

Reduced 

df 

Full 

df 

¥* 

p- value 

Decision 
a  =  0.10 

B 

11727.22 

10635.72 

110 

65 

0.1634 

1 

Do  not  reject  Ho 

2 

60329.88 

23474.3 8 

110 

65 

2.2678 

0.0013 

Reject  Ho  1 

3 

102875 

59517.75 

110 

65 

0.4200 

I 

Do  not  reject  Ho 

m 

56199.44 

32977.03 

110 

65 

1.0172 

0.4687 

Do  not  reject  Ho 

5 

75615.55 

51088.38 

110 

65 

0.6935 

0.9022 

Do  not  reject  Ho 

6 

133235.7 

87235.61 

110 

65 

0.7617 

0.8321 

Do  not  reject  Ho 

Table  3.5  General  Linear  Test  for  Interaction  Terms,  No-Attack  Case 


h 

Reduced 

SSE 

Full 

SSE 

Reduced 

df 

Full 

df 

F* 

p- value 

Decision  | 

a  =  0.10  | 

B 

3124.734 

1865.01 

110 

65 

0.9757 

0.5290 

Do  not  reject  Ho  | 

2 

14753.2 

110 

65 

0.8666 

0.6918 

Do  not  reject  Ho 

3 

18090.36 

10431.2 

■S3 

65 

0.4088 

Do  not  reject  Ho 

B 

19856.42 

12645.76 

110 

65 

0.8236 

0.7528 

Do  not  reject  Ho 

5 

14086.2 

110 

65 

0.8328 

0.7401 

Do  not  reject  Ho 

6 

15461.59 

6518.31 

110 

65 

1.9818 

0.0058 

Reject  Ho 

3.6.2  Purpose  of  Variable  Reduction.  As  we  have  already  pointed 
%  out,  each  daily  metamodel  potentially  contains  62  variables  and  an  intercept 

term.  If  we  disregard  the  seven  blocking  terms  because  they  are  forced  into 
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all  models,  55  candidate  variables  remain  for  possible  inclusion  into  each 
daily  metamodel.  It  should  be  intuitively  clear  that  not  all  55  variables  can 
be  equally  significant  in  helping  to  explain  the  behaviour  of  the  simulation 
model,  and  indeed  some  variables  may  be  entirely  insignificant.  Even  a 
brief  glance  at  preliminary  models  containing  all  possible  variables  strongly 
supports  this  intuition,  as  some  variables  or  interactions  affect  the  number 
of  sorties  flown  by  twenty  to  forty  sorties,  while  many  others  have  an  effect 
of  much  less  than  one  sortie.  The  high  p-values  (probability  of  the  reported 
value  of  the  F  statistic  for  that  variable  occurring  by  chance)  associated 
with  many  of  the  variables  also  indicate  that  those  variables  are 
insignificant.  Obviously  then,  we  are  able  to  reduce  each  metamodel  from  a 
model  containing  all  variables  to  a  model  containing  fewer  more  significant 
variables.  Such  a  reduction  is  desirable  for  several  reasons.  First,  the 
fewer  variables  overall  that  appear  ;u  an  explanatory  model,  the  easier  it  is 
to  understand  the  relationships  between  sorties  generated,  the  independent 
variables,  and  their  interactions.  Second,  as  more  interactive  terms  are 
included,  the  occurrence  of  effects  interacting  significantly  with  more  than 
one  other  effect  increases,  further  complicating  the  analysis.  Third,  as 
Hocking  (1976:7)  states,  a  "motivation  for  variable  elimination  is  that 
smaller  variance  is  achieved  with  a  subset  model,  although  at  the  expense 
of  some  bias  in  the  estimate."  Variable  reduction  is  therefore  important  for 
both  the  explanation  and  prediction  purposes  of  a  metamodel.  For 
explanation  we  seek  to  eliminate  variables  so  that  the  drivers  of  airbase 
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performance  stand  out,  allowing  resource  allocation  to  be  concentrated 
where  it  will  have  the  most  effect;  and  for  prediction  we  also  seek  to 
eliminate  variables  so  that  we  can  estimate  the  sorties  flown  with  greater 
precision. 

a  fi  a  Variant  for  Pmdiefcinn  and  F.atimatinn  Miller  (1990:4-6) 
shows  that  the  variance  of  the  response  decreases  with  fewer  predictors 
included,  and  that  bias  in  the  estimator  for  the  response  increases,  but 
points  out  that  "if  a  variable  has  no  predictive  power,  then  adding  that 
variable  merely  increases  the  variance."  The  dear  implication  is  that  if  a 
variable  has  little  effect  on  the  value  of  the  response,  it  should  be  removed 
to  reduce  the  variance  of  the  estimate.  The  difficulty  is  in  finding  the  right 
tradeoff  between  the  number  of  variables,  variance,  and  bias.  Miller's 
remarks  are  made  in  the  context  of  estimating  the  expected  value  of  a 
response,  so  the  only  component  of  variance  is  the  variance  of  the 
predictors.  The  tradeoff  is  complicated,  however,  if  instead  of  confidence 
intervals  for  the  expected  value  of  the  response,  prediction  intervals  for  a 
single  predicted  value  are  desired.  Equation  3.6  showed  that  for  prediction, 
MSE  must  be  added  to  the  sum  of  the  variances  of  the  predictors.  For  the 
present  data,  MSE  is  substantially  larger  than  the  sum  of  the  variances  of 
the  predictors,  and  does  not  continue  to  decrease  as  more  variables  are 
removed  from  a  model.  Table  3.6  illustrates  the  differences  in  MSE, 
prediction  variance  (s2CYnwr)),  and  estimation  variance  (s3(Yhat»  for  the  full 
model  and  two  reduced  models  on  days  five,  ten,  and  twenty  for  the  no- 


attack  case.  The  first  reduced  model  includes  variables  significant  at  the 
0.10  level,  and  the  second  includes  variables  significant  at  the  0.05  level. 
Similar  results  are  observed  for  all  other  days  across  both  cases.  The  table 
shows  that  while  MSE  for  the  larger  reduced  model  is  always  less  than  for 
the  full  model,  MSE  then  increases  when  more  variables  are  removed.  The 
table  also  shows  that  for  the  reduced  models  MSE  is  dominant  in  the 
calculation  of  variance  for  a  new  prediction,  so  that  the  decrease  in  variance 
due  to  fewer  predictors  is  more  than  offset  by  the  increase  in  MSE. 


Table  3.6  MSE,  Estimation  Variance,  and  Prediction  Variance 


Day 

Full  Model 
(66  Terms) 

Reduced  Model,  10% 

Reduced  Model,  5% 

S 

s* 

(That) 

s* 

<Y-> 

Terms 

^22 

s* 

(Yhat) 

s* 

(Y-J 

Terms 

MSE 

s* 

(Yhat) 

s2 

5 

786 

387 

1173 

10 

t _ 

ia 

78 

cdfi 

ooo 

6 

636 

65 

701 

■ 

10 

943 

464 

1407 

a 

746 

73 

819 

5 

778 

73 

861  I 

D 

635 

263 

798 

13 

426 

66 

492 

9 

478 

59 

jlJ 

3.6.4  Variable  Reduction  and  Bias.  Neter  et  al.  (1989:412)  point  out 
that  it  is  sometimes  preferable  to  have  a  small  amount  of  bias  if  the 
variance  can  be  reduced  enough  that  the  probability  of  a  biased  estimate 
being  closer  to  the  true  value  is  higher  than  the  same  probability  for  an 
unbiased  but  much  higher  variance  estimate.  To  illustrate  this  concept 
consider  the  data  for  day  ten  in  Table  3.6.  We  assume  under  some  set  of 
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inputs  that  the  full  model  produces  an  unbiased  estimate  of  the  expected 
value  of  sorties  flown,  i.e.  Yhat^,,  which  we  shall  assume  to  be  140  sorties, 

•  with  a  variance  sa(Yhatton)  of  464  (from  Table  3.6),  and  a  normal 
distribution.  Assume  that  under  the  same  input  conditions  the  10% 

*  significance  reduced  model  estimates  only  135  sorties,  i.e.  Yhat^  =  135,  for 
a  bias  of  five  sorties.  The  reduced  model  estimate  also  has  a  normal 
distribution,  but  its  variance  s2(YhatfcilJ  is  only  73  (also  from  Table  3.6). 
Both  distributions  are  plotted  in  Figure  3.5,  with  the  lower  curve  the 
distribution  for  the  full  model.  From  Figure  3.5,  the  area  under  the 
respective  curves  between  the  vertical  lines  represents  the  probability  that 
an  estimate  from  that  distribution  will  fall  with  the  range  of  the  true  value 
indicated  by  the  lines.  In  the  example  above,  the  probability  of  the 
unbiased  estimate  falling  within  plus  or  minus  ten  of  the  assumed  true 
value  of  140  is  0.358,  while  the  probability  of  the  biased  estimate  falling 
within  the  same  range  is  0.681.  The  biased  estimator  in  this  case  is 
preferable  because  it  is  more  likely  to  estimate  the  true  value  of  sorties 
flown.  The  difficulties  in  applying  this  concept  are  in  deciding  how  much 
bias  is  acceptable,  and  how  precisely  we  need  to  estimate  the  true  value  of 
sorties  flown.  The  issue  of  bias  for  reduced  models  is  further  addressed 
when  alternate  models  are  evaluated. 

3.6.5  Automated  Selection  of  Variables.  One  of  the  most  difficult 
decisions  when  developing  regression  models  from  a  set  of  independent 
variables  and  their  interaction  terms  is  the  selection  of  which  terms  to 
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Figure  3.5  Typical  Distribution  of  Expected  Value  of  Sorties  Flown 


include  in  the  model  and  which  to  exclude  (Neter  at  al,  1989:437).  A 
number  of  automated  computer  selection  procedures  are  available,  including 
the  backward  elimination  used  by  Diener  to  derive  the  original  models.  The 
backward  procedure  starts  with  all  possible  predictors  included,  and  drops 
predictors  that  make  less  than  a  minimum  contribution  to  the  explanation 
of  the  total  variance.  Diener's  criterion  was  a  significance  level  of  0.10 
(Diener,  1989:74).  Another  automatic  method  is  forward  selection,  a 
technique  generally  favoured  by  Neter  et  al.  (1989:458).  The  forward 
procedure  starts  with  no  predictors  included,  and  adds  only  those  predictors 
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that  make  more  than  a  minimum  contribution  to  the  explanation  of 
variance  (Neter  et  al.,  1989:453,458).  A  third  automatic  procedure  is 
stepwise  regression,  which  combines  both  forward  selection  and  backward 
elimination  by  examining  the  significance  of  the  variables  included  in  the 
model  after  the  addition  of  each  variable,  and  removing  any  variable  which 
no  longer  satisfies  a  minimum  significance  criterion.  For  all  techniques  the 
setting  of  significance  levels  is  a  subjective  assessment  which  will  be  further 
discussed  shortly. 

3-fl.fi  Maiunum  Ra  An  automatic  technique  that  differs  from  the 
previous  three  is  the  maximum  R2  improvement  technique,  which  builds 
models  by  comparing  the  variables  included  within  a  model  of  a  given  size 
with  those  not  in  the  model,  and  switches  them  until  the  model  with 
maximum  R2  for  the  given  size  is  obtained.  The  procedure  is  repeated  for 
all  sized  models,  resulting  in  the  one  variable,  two  variable,  etc  models  with 
maximum  R2  .  The  SAS  User's  Guide:  Statistics  (SAS  Institute,  1985:765) 
states  that  the  maximum  Ra  improvement  technique  "is  considered  superior 
to  the  stepwise  technique  and  almost  as  good  as  all  possible  regressions." 


3.6.7  fifil 


of  Significance  Level  for  Stepwise  Regressions.  In 


stepwise  regression,  the  inclusion  or  removal  of  variables  is  a  series  of 
partial  F  tests  at  the  chosen  significance  1  vel.  The  SAS  Users  Guide: 
Statistics  (SAS  Institute,  1985:765)  points  out  that  "when  many  significance 
tests  are  performed,  each  at  a  level  of,  say  5%,  the  overall  probability  of 
rejecting  at  least  one  true  null  hypothesis  is  much  larger  than  5%."  The 
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null  hypothesis  in  each  case  is  that  the  value  of  the  parameter  for  the 
variable  under  consideration  is  zero,  so  the  consequence  of  rejecting  a  true 
null  hypothesis  (Type  I  error)  is  the  inclusion  of  an  additional  variable  that 
is  not  actually  significant.  This  observation  is  highly  relevant  to  this 
research  because  over  the  first  six  days,  for  both  attack  and  no-attack  cases, 
an  average  of  41  terms  is  rejected  during  the  backward  elimination,  i.e.  41 
significance  tests  are  carried  out.  Forward  selection  should  be  less  prone  to 
such  error  because  for  the  same  significance  level  we  expect  fewer  steps  to 
be  carried  out.  The  implication  of  the  higher  than  expected  probability  of 
Type  I  error  is  that  we  "should  specify  a  very  small  significance  level"  (SAS 
Institute,  1985:766).  The  decision  on  what  constitutes  a  "very  small 
significance  level,"  however,  is  left  to  the  researcher.  For  this  research,  5% 
and  1%  will  be  examined  as  small  significance  levels.  As  an  alternative  to 
basing  inclusion  or  exclusion  of  a  predictor  on  significance  level,  Neter  et  al. 
suggest  a  technique  whereby  variables  are  considered  in  terms  of  their 
marginal  contribution  to  error  reduction.  For  example,  if  in  the  forward 
stepwise  procedure  the  value  of  the  F  statistic  required  for  a  variable  to 
enter  the  model  is  set  at  2.0,  the  effect  is  that  "the  marginal  error  reduction 
associated  with  the  added  variable”  is  "at  least  twice  as  great  as  the 
remaining  error  mean  square  once  that  variable  has  been  added"  (Neter  et 
al.,  1989:457).  Although  the  SAS  software  does  not  use  the  value  of  F  for 
variable  selection,  the  value  of  F  for  each  variable  included  is  reported  in 
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the  output  of  the  automatic  procedures,  so  an  assessment  of  marginal  error 
reduction  can  be  readily  made. 

3.6.8  Best  Subsets  Techniques.  Only  one  model  results  when 
stepwise  regression  techniques  are  used  to  automatically  select  the  terms  to 
be  included  in  a  regression  model,  but  it  is  important  to  recognise  that  this 
model  is  only  one  of  many  possible  models  that  could  have  been  derived.  A 
different  technique  is  to  develop  all  the  possible  models  and  select  horn  that 
set  the  preferred  model.  In  practice,  only  a  subset  of  models  that  meet 
certain  criteria  are  developed  and  examined.  The  criteria  can  be  to  include 
just  the  n  best  models  regardless  of  size,  or,  more  commonly,  to  include  the 
m  best  models  for  a  range  of  model  sizes,  i.e.  number  of  predictors  included. 
The  algorithms  used  to  produce  the  subsets  are  known  as  "best  subsets 
algorithms"  (Neter  et  al.,  1989:462),  and  an  example  is  the  RSQUAKE 
procedure  provided  by  SAS  (SAS  Institute,  1985:711-724).  Without  such 
algorithms,  the  technique  becomes  unworkable  as  the  number  of 
independent  variables  increases.  For  example,  to  examine  all  possible 
regression  models  for  the  data  used  in  this  research  would  entail  examining 
approximately  3.6  x  1016  models  (ten  independent  variables  leads  to  45 
interaction  terms,  for  a  total  of  55  candidate  terms,  and  therefore  2“  =  3.6  x 
10 16  possible  regression  models).  Even  with  an  efficient  algorithm,  which 
uses  branch  and  bound  methods  to  evaluate  only  a  fraction  of  the  possible 
models,  an  inordinate  amount  of  computing  time  may  be  required.  Also,  in 
considering  a  model  of  such  complexity  as  this,  distinguishing  between 


alternatives  offered  in  a  best  subset  may  be  impractical.  The  main 
advantage,  however,  of  using  a  best  subset  approach  instead  of  proceeding 
directly  to  a  stepwise  regression  is  that  the  subset  may  include  a  range  of 
models  that  differ  greatly,  but  have  similar  values  for  R2.  This  allows  the 
modeler  to  more  easily  use  some  judgement  in  balancing  goodness  of  fit  and 
the  level  of  significance  desired  of  a  model  with  the  simplicity  and  ease  of 
understanding  of  the  model.  A  similar  result  could  possibly  be  achieved  by 
carrying  out  many  stepwise  regressions,  but  subtle  differences  between 
possible  models  could  be  missed  unless  the  criteria  for  each  stepwise 
regression  were  very  similar. 

3.6.9  Choice  of  Models  from  a  Subset.  When  a  best  subsets 
technique  yields  several  models  from  which  to  choose,  the  selection  of  the 
preferred  model  is  still  far  from  straightforward.  Several  criteria  can  be 
used,  including  Ra,  adjusted  R2,  and  Mallow’s  Cp  statistic. 

3.6.9. 1  Ra  Criterion.  By  selecting  all  p  -  1  predictors  that  are 
significant  at  the  chosen  level,  forward  selection  implicitly  maximises  R,,2, 
calculated  as  follows: 

-  1  -  (3.8) 

SSTO 

where  SSEp  is  the  residual  (or  error)  sum  of  squares  remaining  when  p  -  1 
predictors  (and  the  intercept,  for  a  total  of  p  parameters)  are  included  in  the 
model,  and  SSTO  is  the  total  sum  of  squares  (Neter  et  al.,  1989:444).  When 
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selecting  possible  models  from  a  subset,  R2  can  also  explicitly  be  used  as  a 
selection  criterion.  An  important  characteristic  of  R2,  however,  is  that  it 
*  reaches  its  mmimnm  only  when  all  predictors  are  included  in  a  model,  so 
some  degree  of  subjective  evaluation  is  required  when  using  this  criterion. 

4  A  balance  between  small  marginal  increases  in  R2  and  the  inclusion  of 
additional  variables  must  be  found  (Neter  et  al.,  1989:444-445). 

3.6.9.2  Adjusted  Ra  Criterion.  The  adjusted  R2  criterion 
attempts  to  make  more  objective  the  assessment  of  whether  to  include  extra 
variables  which  may  have  no  real  predictive  or  explanative  power  in  order 
to  increase  R2  by  a  small  amount  Adjusted  R2  takes  into  account  the  loss 
in  degrees  of  freedom  as  additional  terms  are  added  to  the  model,  and 
unlike  R2,  can  reach  a  maximum  and  decline  as  extra  terms  are  added  that 
do  not  make  a  sufficient  contribution  to  the  explanation  of  the  overall 
variance  to  offset  the  fewer  degrees  of  freedom.  The  selection  criterion  is 
therefore  to  maximise  adjusted  R2.  Adjusted  R2  (R£)  is  calculated  as  follows: 


SSEd 

SSTO 


(3.9) 


where  n  is  the  number  of  observations  in  the  data  set  and  p  the  number  of 
parameters  included  in  the  model  (Neter  et  al.,  1989:446).  We  have  already 
seen,  however,  that  the  large  total  degrees  of  freedom  that  exist  in  models 
derived  from  the  data  set  can  mask  effects  that  would  be  obvious  if  fewer 
variables  were  possible  candidates. 
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3.6.9.3  Mallow’s  Cr  Statistic.  Mallow’s  Cp  statistic  can  also  be 
used  to  assist  in  the  process  of  model  reduction,  and  has  the  desirable 
property  of  suggesting  the  choice  of  subset  models  that  are  relatively 
unbiased.  The  Cp  statistic  considers  mean  squared  error,  which  measures 
the  combined  effect  of  sampling  variance  and  bias  in  an  estimator  of  a 
parameter  (Neter  et  al.,  1989:412).  Cp  is  calculated  as  follows: 


„  SSEa  ,  .  . 

C„m - £  -  (n  -  2p) 

p  MSE 


(3.10) 


* 


* 


where  SSEp  is  the  error  sum  of  squares  for  the  subset  model  with  p 
parameters,  MSE  is  the  mean  square  error  for  the  full  model,  and  n  is  the 
number  of  observations.  A  subset  model  is  considered  a  likely  candidate  for 
acceptance  when  Cp  approaches  p. 

3.6.10  Yates’  Algorithm.  Yates’  algorithm  (Box  et  al.,  1978:323)  is  an 
alternate  technique  to  least  squares  regression  for  deriving  a  regression 
model,  but  applies  specifically  to  full  factorial  experiments,  and  is  therefore 
not  applicable  in  this  research. 

3.6.11  Transformations.  When  the  relative  difference  between  the 
smallest  and  largest  values  of  the  response  is  large,  Box  et  al.  suggest  that 
a  transformation  of  the  response  may  be  appropriate  (1978:334).  Such 
transformations  may  involve  taking  the  inverse  of  the  response,  the  natural 
or  base  10  logarithms  of  the  results,  or  the  square  root  of  the  results  (Neter 
et  al.,  1985:138).  Two  problems  exist  with  transforming  the  response  for 
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this  data  set.  First,  the  experimental  design  presupposes  a  linear 
relationship  between  the  independent  variables  and  the  response;  and 
,  second,  the  occurrence  of  zero  values  for  the  response  precludes  inverse  or 

logarithmic  transformations.  A  brief  examination  of  a  logarithmic 
«  transformation,  achieved  by  setting  zero  responses  to  small  positive  values, 

confirmed  that  such  a  transformation  is  inappropriate.  Although  reasonable 
models  appeared  to  result,  examination  of  the  residuals  showed  serious 
departures  from  normality,  and  strong  patterns  in  the  residual  plots. 
Transformations  are  therefore  not  given  further  consideration. 

3.7  Applicability  of  Regression  Techniques 

This  section  presents  the  initial  results  of  applying  to  Diener's  data 
set  the  techniques  and  considerations  outlined  in  the  previous  section. 

Some  unexpected  results  are  observed,  and  the  reasons  for  the  results  and 
their  impact  on  model  development  are  discussed. 

3.7.1  Comparison  of  Automated  Techniques.  To  provide  a  baseline 
for  comparison,  a  set  of  metamodels  was  developed  for  both  the  attack  and 
no-attack  cases  using  the  same  criteria  specified  in  Diener's  research,  that 
is,  backward  elimination  of  variables,  with  variables  retained  in  the  model 
*  at  a  significance  level  of  0.10.  SAS  Version  6.07  was  used  in  developing 

these  and  all  other  models.  For  initial  comparisons,  only  the  first  six  daily 
'  models  were  calculated.  Models  were  then  calculated  using  both  forward 

selection,  at  a  significance  level  of  0.10,  and  stepwise  selection,  with 
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variables  entering  the  model  at  0.10  significance,  and  remaining  in  the 
model  also  at  0.10  significance.  Comparison  of  the  new  sets  of  models  with 
the  baseline  models  provided  the  first  unexpected  result  in  that  each  set  , 

was  identical,  regardless  of  the  technique  used.  Overall  model  significance, 

SSE,  and  individual  parameter  significance  were  also  identical  for  * 

equivalent  models.  Further  analysis  of  the  regression  results  showed  that 
for  the  stepwise  regression,  none  of  the  variables  was  removed  from  any 
model  after  entering. 

3.7.2  Expected  Results.  The  results  described  in  the  previous 
paragraph  are  surprising  because  the  regression  texts  consulted  create  the 
expectation  that  different  regression  techniques  will  lead  to  different 
models,  and  that  it  is  common  for  variables  to  leave  a  model  as  other 
combinations  of  variables  are  able  to  explain  more  of  the  variance.  Neter  et 
al.  (1989:454-457)  provide  an  example  where  the  first  variable  that  entered 
was  eventually  dropped.  Devore  (1991:550)  summarises  the  usual  approach 
by  stating  that  "a  single  variable  may  be  more  strongly  related  to  y  than 
either  of  two  or  more  other  variables  individually,  but  the  combination  of 
these  variables  may  make  the  single  variable  subsequently  redundant." 

3.7.3  Comparison  Against  Original  Design  Matrix.  As  a  check  that 
the  regressions  had  been  carried  out  correctly,  the  forward  and  stepwise 
techniques  were  applied  to  the  same  data  set,  but  with  the  original  (0,1) 

coded  design  matrix  restored.  Identical  total  stuns  of  squares  using  either  ’ 

design  matrix  indicated  that  no  data  points  had  been  omitted  or  changed; 
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however,  using  the  original  design  matrix  gave  regression  results  more  in 
line  with  expectation,  with  different  regression  techniques  resulting  in 
different  models.  For  example,  on  day  six  of  the  attack  case,  the  first 
variable  selected  (and  therefore  retained)  by  the  forward  selection  technique 
was  also  first  selected  by  the  stepwise  technique,  but  subsequently  dropped. 
The  same  variable  was  also  dropped  by  the  backward  elimination  technique, 
which  returned  a  model  the  same  as  in  Diener's  original  work,  thus 
confirming  that  the  data  had  not  been  altered.  The  reason  for  the 
unexpected  results  was  found  in  an  examination  of  the  various  types  of 
sums  of  squares  that  arise  during  multiple  regression. 

3.7.4  Types  of  flume  of  Squares.  Freund  and  Little  (1985:103-105) 
outline  the  different  sums  of  squares  (SS)  that  are  relevant  in  analysis  of 
variance,  describing  Type  I,  Type  II,  Type  HI,  and  Type  IV  sums  of  squares. 
Neter  et  al.  (1989:271-280)  provide  a  similar  analysis,  referring  to  extra 
sums  of  squares,  but  Freund  and  Little's  description  is  preferred  because 
their  terminology  relates  directly  to  SAS  outputs.  Type  I,  or  sequential  SS 
are  the  sums  of  squares  for  each  predictor  that  result  when  predictors  are 
added  sequentially  to  a  model,  and  can  be  considered  as  the  reduction  in 
SSE  when  a  predictor  is  added,  given  that  all  previous  predictors  are 
already  in  the  model.  Type  I  SS  are  therefore  dependent  upon  the  order  in 
which  terms  are  added  to  a  model.  Type  II  SS  for  a  given  variable  are 
calculated  considering  the  effect  of  all  other  predictors  in  the  model  that  do 
not  contain  the  effect  of  that  variable.  Type  III  SS  are  referred  to  as  partial 


sums  of  squares,  and  represent  the  reduction  in  SSE  due  to  a  predictor 
given  that  all  other  predictors  being  considered  are  already  present  in  the 
model.  Type  IV  SS  are  intended  for  analysis  when  the  design  matrix  has 
empty  cells. 

3.7.5  Experimental  Design  and  Sums  nf  Snnarea.  The  experimental 
design  chosen  by  Diener  is  characterised  by  just  one  observation  at  each 
design  point.  For  example,  the  low  level  of  factor  D,  and  the  high  level  of 
all  other  factors  occurs  once  in  the  design  matrix,  as  do  all  the  other  127 
factor  combinations  modeled  (Diener,  1989:45).  Freund  and  Little 
(1985:106)  refer  to  this  data  structure  as  equal  cell  frequencies,  and  state 
that  with  such  a  data  structure,  all  types  of  sums  of  squares  (SS)  will  be 
equal  for  any  predictor,  whether  a  main  effect  or  an  interaction.  Use  of  the 
SAS  General  Linear  Models  procedure  confirmed  that  for  (-1,1)  coding,  each 
predictor  (including  the  interaction  terms)  has  all  four  types  of  SS  equal  to 
each  other.  To  understand  the  effect  on  regression  of  having  predictors  with 
equal  Type  I  and  Type  III  SS,  consider  a  model  with  only  three  possible 
predictors:  Xx,  X,,  and  Xg.  Assume  also  that  the  predictors  will  be  entered 
in  numeric  order.  Referring  to  the  previous  paragraph,  the  Type  I  SS  for  X! 
is  the  reduction  in  SSE  due  to  Xx  alone,  as  no  other  predictors  have  yet  been 
added.  The  Type  HI  SS  for  Xlt  however,  is  the  reduction  in  SSE  due  to  Xx, 
given  that  Xg  and  Xg  are  already  in  the  model.  If  the  Type  I  and  ID  SS  are 
equal,  having  Xg  and  Xg  already  in  the  model  has  no  effect  on  the 
contribution  that  Xx  makes.  To  summarise,  if  the  Type  I  and  III  SS  are 
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equal  for  each  predictor,  then  it  is  dear  that  each  predictor  makes  a 
contribution  to  the  reduction  of  SSE  completely  independently  of  its  order  of 
entry  and  the  presence  of  any  other  predictors. 

3-7-6  Explanation  nf  Regression  Results.  The  equivalence  of  forward 
selection,  backward  elimination,  and  stepwise  selection  in  developing  models 
is  readily  explained  once  we  realise  that  the  contribution  to  the  explanation 
of  variance  that  a  variable  makes  does  not  vary  depending  upon  whether  it 
is  the  only  variable  in  a  model  or  the  last  of  many.  Stepwise  selection 
becomes  equivalent  to  forward  selection  because  the  introduction  of 
subsequent  variables  does  not  change  the  sum  of  squares  contribution  of  a 
variable  already  in  the  model.  If  their  sums  of  squares  do  not  reduce,  their 
significance  levels  will  not  reduce,  so  no  variables  will  ever  be  eliminated. 
Forward  and  backward  selection  are  equivalent  because  the  variables  can 
be  ordered  by  their  contribution  to  the  model,  and  the  order  does  not  change 
as  the  model  size  changes.  Therefore  the  same  subset  of  variables 
significant  at  a  given  level  will  result  whether  we  start  with  the  most 
important  variable  and  add  variables,  or  start  with  all  variables  and  delete 
the  least  significant.  An  interesting  observation  is  that  when  a  variable  is 
added  to  a  model,  the  significance  of  variables  already  in  the  model  usually 
increases  because  the  Type  II  SS  for  variables  in  the  model  remains 
constant,  but  the  overall  SSE  and  therefore  MSE  reduce,  so  that  F  (equal  to 
Type  II  SS/MSE)  increases.  This  observation  holds  as  long  as  the  variable 


being  added  reduces  SSE  enough  to  offset  the  loss  of  a  degree  of  freedom, 
and  thus  reduces  MSE. 

3.7.7  Other  Unexpected  Results.  Comparison  of  the  full  models  and 
reduced  models  used  to  carry  out  the  General  Linear  Tests  revealed  that, 
for  both  the  full  and  reduced  models,  the  value  of  the  intercept  estimate 
remained  equal  to  the  mean  number  of  sorties  generated  by  the  128 
treatments  for  any  given  day  even  though  55  interaction  terms  were  not 
included  in  the  reduced  models.  Additionally,  the  parameter  estimates  of 
the  main  terms  remaining  in  the  reduced  models  were  the  same  as  for  the 
main  terms  in  the  full  models.  Comparing  the  baseline  10%  significance 
models  with  the  full  and  main  effects  model  gave  the  same  result,  that  is, 
the  estimate  for  a  particular  parameter  in  any  given  model  does  not  change, 
regardless  of  the  number  or  combination  of  the  other  terms  in  that  model. 
The  extreme  example  is  the  first  daily  model  in  the  attack  case.  Figure  3.6 
contains  part  of  the  analysis  of  variance  table  for  the  full  model  on  day  one, 
with  all  the  interaction  terms  omitted  for  clarity,  and  Figure  3.7  contains 
the  full  analysis  of  variance  table  for  the  10%  significance  model  on  day  one. 
Comparison  of  the  two  figures  shows  that  the  intercept  and  variable  D 
estimates  are  identical  for  both  the  reduced  and  full  models,  even  though 
the  full  model  contains  all  55  variables,  and  the  10%  significance  model 
contains  just  one  variable.  The  independence  of  parameter  estimates  with 
regard  to  other  variables  in  the  model  is  a  similar  effect  to  the 
independence  of  each  variable's  contribution  to  sums  of  squares  explained 
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earlier.  An  analogy  with  simple  linear  regression  shows  why  the  intercept 
estimate  does  not  change,  while  examination  of  some  of  the  properties  of  the 
design  matrix  helps  explain  why  the  parameter  estimates  are  independent 
of  each  other,  and  also  provides  the  underlying  reason  for  the  equality  of 
the  different  types  of  sums  of  squares. 


Full  Model  Attack  Case 


Analysis  of  Variance 


Sum  of  Mean 


Source 

DF  Squares  Square  F  Value 

Prob>F 

Model 

62  30882.75000  498.10887  3.073 

0.0001 

Error 

65  10535.71875  162.08798 

C  Total 

127  41418.46875 

Root  MSE 

12.73138 

R-square 

0.7456 

Dep  Mean 

89.60938 

Adj  R-sq 

0.5030 

C.V. 

14.20764 

Parameter  Estimates 

Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter»0 

Prob  >  I T | 

INTERCEP 

1 

89.609375 

1.12530545 

79.631 

0.0001 

B1 

1 

-10.171875 

2.97727836 

-3.417 

0.0011 

B2 

1 

3.453125 

2.97727836 

1.160 

0.2504 

B3 

1 

-13.359375 

2.97727836 

-4.487 

0.0001 

B4 

1 

2.828125 

2.97727836 

0.950 

0.3457 

BS 

1 

16.828125 

2.97727836 

5.652 

0.0001 

B6 

1 

8.703125 

2.97727836 

2.923 

0.0048 

B7 

1 

-2.609375 

2.97727836 

-0.876 

0.3840 

A 

1 

-0.125000 

1.12530545 

-0.111 

0.9119 

B 

1 

-0.203125 

1.12530545 

-0.181 

0.8573 

C 

1 

0.203125 

1.12530545 

0.181 

0.8573 

D 

1 

11.953125 

1.12530545 

10.622 

0.0001 

E 

1 

0.750000 

1.12530545 

0.666 

0.5075 

F 

1 

0.109375 

1.12530545 

0.097 

0.9229 

G 

1 

0.062500 

1.12530545 

0.056 

0.9559 

H 

1 

0.421875 

1.12530545 

0.375 

0.7090 

J 

1 

-0.843750 

1.12530545 

-0.750 

0.4561 

K 

1 

0.218750 

1.12530545 

0.194 

0.8465 

JK 

1 

-0.953125 

1.12530545 

-0.847 

0.4001 

Figure  3.6  Analysis  of  Variance,  Day  One  Attack  Case,  Full  Model, 
with  Coefficients  and  Statistics  for  Interaction  Terms  Omitted 


3.37 


Attack  Case,  10% 

Day  1 

Analysis  of  Variance 

Sum  of  Mean 

Source 

DF  Squares  Square  F  Value 

Prob>F 

Model 

8  29484. 

62500  3685.57813  36.751 

0.0001 

Error 

119  11933. 

84375  100.28440 

C  Total 

127  41418. 

46875 

Root  MSE 

10.01421 

R-square 

0.7119 

Dep  Mean 

89.60938 

Adj  R-sq 

0.6925 

C.V. 

11.17540 

Parameter  Estimates 

Parameter  Standard 

T  for  HO: 

Variable 

DP 

Estimate 

Error 

Pararaeter-0 

Prob  >  | T I 

INTERCEP 

1 

89.609375 

0.88513947 

101.238 

0.0001 

B1 

1 

-10.171875 

2.34185892 

-4.344 

0.0001 

B2 

1 

3.453125 

2.34185892 

1.475 

0.1430 

B3 

1 

-13.359375 

2.34185892 

-5.705 

0.0001 

B4 

1 

2.828125 

2.34185892 

1.208 

0.2296 

BS 

1 

16.828125 

2.34185892 

7.186 

0.0001 

B6 

1 

8.703125 

2.34185892 

3.716 

0.0003 

B7 

1 

-2.609375 

2.34185892 

-1.114 

0.2674 

D 

1 

11.953125 

0.88513947 

13.504 

0.0001 

Figure  3.7  Analysis  of  Variance,  Day  One  Attack  Case, 
10%  Significance  Reduced  Model 


3.7.8  Constant  Intercept  Eatimntft.  A  characteristic  of  fractional 
factorial  designs  is  that  each  column  of  the  design  matrix  has  an  equal 
number  of  plus  and  minus  ones  (Box  and  Meyer,  1993:94).  As  each  column 
represents  a  single  variable,  consider  simple  linear  regression  with  just  one 
independent  variable  X,  which  takes  the  values  -1  and  +1,  and  assume  that 
there  are  an  equal  number  of  observations  of  the  dependent  variable  Y  for 
each  level  of  X.  If  there  is  no  relationship  between  X  and  Y,  the  regression 
line  will  be  horizontal,  passing  through  the  mean  of  Y.  If  there  is  some 
linear  relationship,  the  regression  line  will  have  a  slope,  but  the  intercept 
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will  remain  the  mean  of  Y  because  there  are  equal  numbers  of  observations 

at  both  levels  of  X,  i.e.  the  mean  of  X  is  0. 

3.7.9  Design  Matrix  Properties.  Reynolds  (1993)  states  that  when  a 

design  matrix  is  orthogonal,  that  is,  there  is  no  relationship  between  the 

independent  variables,  the  effects  of  the  independent  variables  can  be 

considered  separately  in  a  regression  model.  Neter  et  al.,  while  not 

addressing  orthogonality  as  such,  state  that 

"in  general,  when  two  or  more  independent  variables  are 
uncorrelated,  the  marginal  contribution  of  one  independent  variable 
in  reducing  the  error  sum  of  squares  when  the  other  independent 
variables  are  in  the  model  is  exactly  the  same  as  when  this 
independent  variable  is  in  the  model  alone."  (Neter  et  al.,  1989:298) 

To  test  for  orthogonality,  the  transpose  of  the  design  matrix  is  multiplied  by 

the  design  matrix.  If  the  resulting  matrix  is  diagonal,  that  is,  all  elements 

except  on  the  main  diagonal  are  zero,  then  the  design  matrix  is  orthogonal 

(Reynolds,  1993).  Both  the  (-1,1)  and  (0,1)  matrices  were  tested,  and  the 

(-1,1)  matrix  proved  to  be  orthogonal,  while  the  (0,1)  matrix  did  not.  Tests 

of  correlation  among  the  independent  variables  showed  no  correlation  for 

either  matrix.  Considering  that  Neter  et  al.  specify  a  first  order  model 

(1989:297),  a  possible  reason  for  the  models  derived  using  the  (0,1)  design 

not  behaving  in  the  same  way  as  the  (-1,1)  models,  even  though  none  of  the 

independent  variables  are  correlated,  is  the  inclusion  of  interaction  terms  in 

the  models.  If  the  (0,1)  design  matrix  was  expanded  to  include  interactions, 

the  four  interactions  between  two  variables  ((high, high);  (high,low); 

(low,high);  (low,low))  would  introduce  three  zeros  into  the  matrix  for  every 


3.39 


one,  thus  unbalancing  the  number  of  ones  and  zeros.  For  the  (-1,1)  matrix, 

the  interaction  terms  introduce  an  even  number  of  plus  and  minus  ones.  It 

is  sufficient  for  this  research  that  the  (-1,1)  matrix  produces  results  that  are 

expected.  Further  investigation  of  the  properties  of  the  (0,1)  design  is 

beyond  the  scope  of  this  research.  t 

3.7.10  Resolution  of  Unexpected  Results.  Because  the  effects 
observed  in  deriving  models  from  the  (-1,1)  design  matrix  can  be  explained 
with  reference  to  the  sums  of  squares  of  the  variables,  the  lack  of 
correlation  of  the  variables,  and  the  orthogonality  of  the  design  matrix,  the 
models  are  accepted  as  valid,  and  we  now  examine  the  effect  on  the 
development  of  new  models  of  the  results  discussed  in  the  preceding 
paragraphs. 

3.7.11  Implications  for  Automated  Techniques.  Because  forward 
selection,  backward  elimination,  and  stepwise  selection  all  result  in  the 
same  model  for  a  chosen  significance  level,  the  preferred  technique  is 
forward  selection.  In  general,  many  more  terms  are  left  out  of  a  model  than 
are  included  so  forward  selection  is  substantially  faster  than  backward 
elimination.  Also,  under  the  circumstances  existing  for  this  data  set,  the 
order  of  entry  into  a  model  developed  by  forward  selection  is  an  indication 
of  the  relative  importance  of  the  variables  to  the  explanation  of  variance.  A 
particularly  important  consequence  of  order  independence  of  selection  is 

that  the  automated  techniques  will  be  fully  effective  at  finding,  for  the  » 

chosen  significance  level,  the  model  yielding  the  highest  value  for  R2.  Miller 
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(1990:48-53;  70-75)  points  out  that  this  is  frequently  not  the  case  when  the 
different  types  of  sums  of  squares  are  not  equal  and  provides  several 
examples  in  which  automated  selection  procedures  failed  to  find  the  best 


fitting  subsets  of  varying  sizes.  The  overall  implications  for  this  research 
are  that  we  can  be  confident  of  finding  the  "best"  model  (based  on  R*  and 
SSE)  at  a  given  significance  level  using  forward  selection;  and  that  any 
automatically  produced  model  containing  a  given  number  of  terms  will  have 
the  highest  R3  of  all  models  containing  that  number  of  terms. 


The  models 


developed  by  using  the  maximum  R2  automatic  technique  will  reflect  the 
order  of  entry  of  variables  into  a  model  developed  by  forward  selection.  For 
example,  the  best  five  variable  model  found  using  maximum  R3  will  contain 
the  first  five  variables  chosen  during  forward  selection.  Because  the 
relative  importance  of  the  variables  never  changes,  maximum  R3  will  not 
find  any  "unusual”  combination  of  n  variables  that  result  in  a  higher  R3 
than  the  first  n  variables  ordered  by  their  contribution  to  sum  of  squares. 
The  maximum  R3  technique  is  still  useful,  however,  if  we  wish  to  compare  a 
set  of  models  containing,  say,  8,  9  and  10  variables. 


The  implications 


for  best  subsets  techniques  of  variables  having  independent  effects  are 
similar  to  the  implications  for  maximum  R3  technique.  The  best  subset  of 
size  n  will  always  contain  the  first  n  variables  chosen  by  forward  selection, 
and  the  next  best  subset  will  include  the  n  •  1  most  significant  variables, 
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and  the  n  +  2th  most  significant  For  example,  if  the  three  best  subsets  of 
four  variables  are  requested,  the  best  subset  will  contain  the  first  four 
variables  ordered  by  significance,  the  second  best  subset  will  contain  the 
first  three  and  the  fifth  variables  ordered  by  significance,  while  the  third 
best  subset  will  contain  the  first  three  and  the  sixth  variables  ordered  by 
significance.  Clearly,  there  is  little  to  be  gained  by  requesting  other  than 
the  single  best  subset  of  each  size  because  all  subsets  of  each  size  will  differ 
only  in  the  last  variable,  assuming  that  the  number  of  subsets  examined  is 
smaller  than  the  number  of  variables  not  included.  If  only  the  single  best 
subset  for  a  range  of  model  sizes  variables  is  desired,  then  the  maximum  R2 
technique  is  preferable  because  both  techniques  will  return  the  same  result, 
but  the  maximum  Ra  technique  is  faster  and  is  able  to  cope  with  many  more 
variables. 

3.7.14  -8AS  RSQUARE  Procedure  for  Best  Subsets.  The  RSQUARE 
procedure  provided  in  SAS  for  finding  best  subsets  is  unable  to  handle  all 
55  variables  and  interactions  that  are  possible  candidates  in  this  research. 
However,  for  each  day  we  know  that  the  55  -  n  least  significant  variables 
when  ordered  by  their  sums  of  squares  will  never  enter  a  model  of  size  n,  so 
we  can  safely  reduce  the  pool  of  variables  considered  by  RSQUARE  if  we 
have  some  idea  of  the  maximum  size  of  model  to  be  considered. 

Examination  of  the  10%  significance  models,  and  the  tenns  not  included  in 
those  models  suggests  that  a  conservative  maximum  model  size  is  35  terms 
(the  largest  10%  model  contains  17  terms).  If  the  twenty  least  significant 
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variables  are  deleted  from  the  pool,  the  RSQUARE  procedure  is  able  to 
produce  for  all  thirty  days  a  single  best  subset  of  each  model  size  specified 
in  only  a  few  minutes  processing  time  on  a  VAX  Model  6000-420,  compared 
to  over  24  hours  when  three  subsets  were  requested  and  the  48  most 
significant  variables  were  considered.  An  undesirable  consequence  of 
truncating  the  variable  pool  to  use  RSQUARE  is  that  the  values  reported 
for  Cp  are  not  the  same  as  those  obtained  using  forward  selection.  The 
reason  is  that  the  SAS  procedure  uses  the  full  model  MSE  to  estimate 
variance,  and  reducing  the  variable  ~ool  changes  the  full  model  as  seen  by 
RSQUARE  (SAS  Institute,  1985:715,765).  For  the  situation  in  this 
research,  the  RSQUARE  procedure  for  best  subsets  is  most  applicable  for 
efficiently  presenting  Ra  and  adjusted  Ra  values  for  a  range  of  model  sizes. 

3.7.15  Model  Selection  Criteria.  The  preliminary  use  of  the 
techniques  above  strongly  suggests  that  Mallow’s  Cp  is  of  no  practical  use  in 
selecting  models  from  a  subset  of  candidate  models  because  Cp  frequently 
takes  negative  values.  When  Cp  is  negative  the  recommended  plot  of  Cp 
against  p,  where  p  is  the  number  of  predictors  in  the  model,  is  meaningless. 
The  reason  for  negative  Cp  values  has  not  been  fully  determined,  but  study 
of  Equation  3.10  indicates  that  the  large  number  of  observations  (128) 
relative  to  the  number  of  variables  generally  included  in  the  models  (1-17) 
outweighs  the  ratio  of  SSEp  to  MSE.  Adjusted  Ra  may  be  of  limited 
usefulness  in  developing  explanatory  models  because  it  very  rarely  reaches 
a  mflrimiim  until  more  than  20  terms  are  included.  However,  maximum 
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adjusted  Ra  is  equivalent  to  minimum  MSE  (Neter  et  al.,  1989:446),  and 
may  well  be  useful  for  finding  models  that  are  most  suitable  for  prediction 
because  we  saw  earlier  that  MSE  was  dominant  in  the  expression  for  the 
variance  of  a  new  prediction. 

3.7.16  Selection  Bias.  Miller  (1990:Ch  5)  devotes  a  chapter  to  the 
issue  of  what  he  terms  "selection  bias,"  which  is  bias  in  the  predicted  value 
of  the  response  resulting  from  using  the  same  set  of  data  for  both  the 
selection  of  a  subset  of  predictor  variables,  and  the  estimation  of  the  value 
of  the  regression  coefficients  for  those  predictors.  According  to  Miller 
(1990:12),  the  bias  results  from  the  fact  that  the  regression  coefficients  of  a 
subset  of  predictors  are  conditional  on  the  subset  chosen.  That  is,  the 
regression  coefficients  of  the  variables  included  in  a  reduced  model  will  be 
different  from  the  coefficients  of  the  same  variables  in  the  full  model.  In 
the  ideal  case,  he  suggests  that 

an  independent  sample  should  be  obtained  to  test  the  adequacy  of  the 
prediction  equation.  Alternatively  the  data  set  may  be  divided  into 
three  parts;  one  part  to  be  used  for  model  selection,  the  second  for  the 
calibration  of  parameters  in  the  chosen  model,  and  the  last  part  for 
testing  the  adequacy  of  the  predictions.  (Miller,  1990:13) 

In  the  current  research  we  are  unable  to  split  the  data  as  Miller  suggests, 

and  carrying  out  additional  simulation  runs  to  gather  more  data  is  well 

beyond  the  scope  of  this  research.  We  have  seen,  however,  that  for  this 

data  set  the  regression  coefficients  in  a  reduced  model  do  not  depend  on 

how  the  model  was  selected,  so  selection  bias  can  be  discounted. 
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The  results  of  the  preliminary  exploration  of  model  building 
techniques  discussed  in  the  previous  two  sections  suggest  that  some  revision 
is  required  of  the  techniques  initially  outlined. 

3.8.1  Automated  Techniques.  We  have  seen  that  the  orthogonality  of 
the  design  matrix  and  the  absence  of  correlation  between  the  independent 
variables  results  in  regression  models  in  which  each  variable  can  be 
considered  independently  of  the  others.  If  we  wish  to  develop  models  at  a 
given  significance  level,  then  forward  selection  is  the  most  efficient  method 
to  use,  and  we  know  that  the  best  fitting  model  at  that  significance  level 
will  be  chosen. 

3.8.2  Exploratory  Techniques,  If  we  wish  to  explore  a  range  of 
possible  models  with  no  specific  significance  level  required,  a  number  of 
alternatives  are  available.  The  RSQUARE  procedure  can  be  used  to  develop 
a  set  of  models  for  each  day  up  to  a  specified  size,  or  a  manual  technique 
can  be  used  to  search  for  appropriate  models.  The  main  advantage  over 
forward  selection  of  examining  models  containing  varying  numbers  of  terms 
is  that  there  may  be  several  potential  terms  which  have  significance  levels 
only  slightly  above  or  below  the  selection  criterion  set  for  the  automatic 
method.  For  example,  the  forward  selection  approach  at  a=0.10  significance 
will  exclude  a  term  significant  at  a=0.099,  and  include  a  term  significant  at 
0=0.101,  when  there  is  no  practical  difference  in  significance  between  the 
two.  Ideally,  we  would  like  to  find  some  clear  division  between  the 
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significance  of  the  last  term  included  and  the  significance  of  the  terms 
excluded. 

3A3  Manual  Search  The  independent  effect  of  the  variables  means 
that  prior  to  building  a  reduced  regression  model,  we  can  examine  the 
contribution  that  each  variable  could  make  to  such  a  model  and  determine 
in  what  order  the  variables  will  enter  the  model.  As  a  preliminary  step,  a 
full  model  containing  all  possible  variables  is  calculated  to  provide  a  list  of 
the  sum  of  squares  for  all  variables.  If  the  variables  are  then  sorted  in 
decreasing  order  of  their  sums  of  squares,  we  have  ranked  the  variables  in 
the  order  that  they  will  enter  a  model  (or  reverse  order  that  they  will 
leave),  regardless  of  the  regression  technique  used.  We  can  also  assess  the 
overall  importance  of  the  variables  by  calculating  a  partial  R2  for  each 
variable.  A  manual  search  is  feasible  only  because  the  variables  always 
enter  a  model  in  order  of  their  significance  in  a  full  model.  The  benefit  of 
using  a  manual  procedure  to  search  for  appropriate  models  is  that  it  is 
practical,  once  the  variables  are  sorted,  to  calculate  both  partial  R2  and 
model  R2  for  each  size  model,  so  that  distinctions  between  more  and  less 
significant  groups  of  variables  are  apparent.  A  disadvantage,  which  also 
applies  to  the  RSQUARE  technique,  is  that  once  a  model  has  been  chosen 
another  procedure,  such  as  the  REG  procedure  (SAS  Institute,  1985:655- 
710),  must  be  used  to  obtain  the  detailed  analysis  of  variance  information 
for  that  model. 
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3.9  Development  of  New  Models 


In  developing  potential  new  models,  the  two  aims  of  a  metamodel, 
explanation  and  prediction,  are  addressed  by  initially  developing  separate 
models  for  each  aim.  We  then  examine  the  different  models  to  determine 
whether  an  all-purpose  model  is  an  acceptable  compromise  for  both 
explanation  and  prediction. 

3.10  Models  for  Prediction 

The  first  models  to  be  developed  are  designed  for  prediction,  so  that 
the  specific  characteristics  desired  in  such  models  are  accuracy  in  predicting 
the  number  of  sorties  flown,  and  precision,  which  in  this  case  can  be 
interpreted  as  the  width  of  a  confidence  interval  containing  a  prediction. 
Accuracy  is  equivalent  to  lack  of  bias,  and  precision  is  directly  proportional 
to  variance.  Variance  can  be  readily  assessed  from  the  regression  statistics 
of  a  model,  but  bias  is  much  more  difficult  to  determine.  Bias  may  be 
defined  as  a  consistent  overestimation  or  underestimation  of  a  true  value, 
which  is  unknown  in  a  probabilistic  simulation  experiment  such  as  Diener's 
(1989).  The  best  that  we  can  do  is  to  assume  that  a  full  model  containing 
all  terms  does  not  contain  any  inherent  bias.  Such  an  assumption  is  a 
consequence  of  the  classical  assumptions  for  regression  analysis,  which 
include  zero  expected  value  for  the  error  term  in  a  model.  We  could 
therefore  avoid  the  issue  of  bias  altogether  by  using  a  full  model,  but  as 
Table  3.6  and  Figure  3.5  showed,  the  variance  in  the  estimates  and 


predictions  of  a  foil  model  are  very  high  relative  to  a  reduced  model,  so  a 
foil  model  is  unable  to  satisfy  our  desire  for  precision.  By  deliberately 
removing  terms  from  a  foil  model  to  produce  a  reduced  model,  we  recognise 
that  some  bias  is  introduced,  but  hopefully  the  terms  removed  make  such  a 
am«H  contribution  to  the  model  that  the  bias  is  negligible.  The  reduction  in 
variance,  however,  can  be  substantial,  as  even  the  least  significant  term 
makes  the  same  contribution  to  model  variance  as  the  most  significant.  To 
assess  the  bias  introduced  into  a  reduced  model,  the  number  of  sorties 
predicted  by  a  reduced  model  is  compared  with  the  number  of  sorties 
predicted  by  the  full  model  at  each  of  two  design  points:  all  factors  at  their 
low  level,  and  all  at  their  high  level.  A  possible  weakness  of  testing  for  bias 
in  this  way  is  that  all  the  interaction  terms  have  the  same  effect  on  the 
predicted  response  whether  all  factors  are  high,  or  all  are  low,  but  the  use 
of  other  factor  combinations  is  impractical  because  the  large  number  of 
interaction  terms  leads  to  an  overwhelming  number  of  choices  that  could  be 
tested.  In  most  models  assessed,  however,  the  dominant  terms  are  main 
effects,  which  do  lead  to  a  change  in  predictions  at  the  two  factor  levels. 

3.10.1  Model  Building.  Pour  sets  of  models  were  developed  and 
evaluated  for  their  suitability  for  prediction.  The  first  set  of  models  is  the 
baseline  set,  calculated  using  forward  selection,  with  terms  in  the  models 
significant  at  the  0.10  level.  A  second  set  of  models  was  also  calculated 
using  forward  selection,  but  at  a  significance  level  of  0.05.  For  the  third  set 
of  models,  the  SAS  RSQUARE  procedure  was  used  to  find  the  subset  of 
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variables  for  each  day  that  returned  the  highest  value  of  adjusted  R8  and 
therefore  minimum  MSE,  and  then  those  variables  were  forced  into  models 
•  calculated  by  the  SAS  REG  procedure.  Developing  the  fourth  set  of  models 

involved  an  assessment  of  the  marginal  reduction  in  MSE  as  terms  were 
4  added  to  the  models.  Evaluation  of  the  minimum  MSE  models  showed  that 

the  last  few  terms  added  to  the  model  produced  decreases  in  MSE  much 
smaller  than  the  increase  in  estimation  variance  resulting  from  the 
inclusion  of  an  additional  predictor.  Equations  3.5  and  3.6  show  that  when 
a  term  is  added  to  a  model,  if  MSE  reduces  by  less  than  the  variance  of  that 
term,  the  overall  prediction  variance  will  increase.  The  decision  rule  for 
inclusion  of  terms  involved  examining  the  variance  of  the  predictors  in  the 
models  with  maximum  R8,  and  then  including  only  the  predictors  that 
reduced  MSE  by  at  least  a  minimum  amount.  The  number  of  models  to 
evaluate,  and  the  small  change  in  predictor  variance  as  further  terms  are 
added  dictated  using  an  average  value  as  the  minimum  acceptable  reduction 
in  MSE.  A  value  of  1.8  was  chosen  for  the  attack  case,  and  1.2  for  the  no- 
attack  case.  Applying  the  decision  rule  to  the  variable  subsets  generally 
revealed  a  distinct  cutoff  in  MSE  reduction  at  around  the  values  chosen, 
with  the  variables  excluded  making  a  substantially  smaller  reduction  in 
,  MSE.  In  some  cases  the  difference  between  the  minimum  MSE  and  the 

near  minimum  MSE  models  proved  to  be  only  two  or  three  terms,  while  in 
other  cases,  the  difference  was  ten  or  more  variables. 
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Table  3.7  and  Table  3.8  present  a 


comparison  of  the  sorties  predicted  for  the  attack  case  with  all  factors  high 
and  all  factors  low  respectively  while  Tables  3.9  and  3.10  present  the  same 
comparisons  for  the  no-attack  case.  The  column  labeled  "Bias"  in  each  table 
represents  the  difference  between  sorties  predicted  by  the  reduced  model  4 

and  sorties  predicted  by  the  full  model.  As  an  indicators  of  the  overall 
accuracy  of  each  model,  the  mean  absolute  deviation  from  the  full  model 
prediction  and  the  mean  absolute  percentage  deviation  appear  at  the  bottom 
of  each  table.  Absolute  deviation  is  measured  because  positive  and  negative 
deviations  could  cancel,  so  that  their  mean  would  understate  the  true 
amount  of  inaccuracy  present.  The  variances  for  estimation  of  the  expected 
value  of  sorties  flown  are  presented  in  Table  3.11,  and  the  variances  for  the 
prediction  of  sorties  flown  are  presented  in  Table  3.12. 

3.10.2.1  Bias.  Several  points  are  evident  from  Tables  3.7 
through  3.10.  First,  for  both  the  attack  and  no-attack  case,  the  performance 
of  the  models  containing  terms  significant  at  the  0.05  level  is  worse  than  all 
other  models.  Such  a  result  is  expected  because  these  models  contain  the 
fewest  terms.  The  particularly  poor  performance  of  the  5%  models  at  the 
low  factor  level  in  the  attack  case  suggests  that  they  are  not  suitable  for 
prediction,  so  they  are  excluded  from  further  consideration.  Second,  with  * 

the  exception  of  the  10%  significance  models  for  predicting  sorties  when  all 
factors  are  low  in  the  attack  case,  all  three  remaining  sets  of  models  make  » 
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reasonably  accurate  predictions  compared  to  the  full  models,  with  relative 
errors  typically  less  than  5  percent. 


Table  3.7  Comparison  of  Sorties  Predicted,  Attack  Case,  All  Factors  High 
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Table  3.8  Comparison  of  Sorties  Predicted,  Attack  Case,  All  Factors  Low 


Full 

Model 

10%  Significance 
Reduced  Model 

5%  Significance 
Reduced  Model 

Minimum  MSE 
Reduced  Model 

Day 

Sorties 

Sorties 

Bias 

Sorties 

Bias 

Sorties 

Bias 

75 

78 

2 

78 

2 

77 

34 

45 

11 

40 

6 

39 

78 

88 

10 

92 

14 

83 

57 

71 

14 

85 

28 

59 

5 

40 

67 

27 

82 

42 

44 

6 

107 

132 

25 

155 

48 

88 

-19 

7 

85 

107 

UMI 

124 

39 

100 

15 

8 

93 

108 

15 

118 

25 

103 

■3 

9 

81 

93 

12 

109 

28 

66 

10 

77 

99 

21 

106 

HfUiEIi] 

80 

H 

11 

79 

84 

5 

96 

83 

H 

12 

68 

82 

15 

86 

18 

76 

8 

13 

67 

86 

19 

77 

■ 

14 

67 

84 

-  Id 

88 

21 

63 

Md 

15 

59 

70 

78 

69 

16 

61 

70 

74 

71 

■9 

17 

58 

57 

■a 

57 

■Bn 

61 

3 

18 

62 

59 

u 

62 

■SBl 

56 

-6 

19 

57 

68 

ii 

68 

ii 

62 

5 

58 

66 

8 

69 

ii 

56 

-2 

21 

55 

55 

58 

3 

54 

Kj 

43 

49 

52 

10 

44 

Mi 

Mill 

43 

46 

49 

6 

43 

SSH 

24 

38 

52 

14 

60 

Il'Cj:  <VV.] 

43 

5 

25 

50 

18 

50 

18 

32 

0 

26 

29 

42 

13 

47 

19 

31 

2 

27 

31 

34 

3 

39 

8 

28 

-3 

28 

26 

29 

3 

34 

8 

28 

2 

29 

28 

33 

6 

36 

8 

29 

m 

30 

29 

24 

-5 

33 

4 

25 

Mean 

1  Error  1 

11.3 

17.0 

MAPE 

20.9% 

30.3% 

9.0% 

Near  Min  MSE 


Reduced  Model 


Sorties 

Bias 

78 

2 

43 

9 

88 

HE 

66 

ME 

44 

He 

91 

-16 

98 

13 

100 

8 

68 

13 

3 

78 

-1 

78 

10 

77 

10 

72 

5 

73 

14 

75 

15 

61 

3 

56 

-6 

70 

13 

56 

-2 

50 

-5 

39 

-4 

46 

3 

42 

3 

25 

37 

37 

31 

29 

26 

7.1 

13.2% 


3.52 


Table  3.9  Comparison  of  Sorties  Predicted,  No-Attacks,  All  Factors  High 


Minimum  MSE 
Reduced  Model 

Sorties 

Bias 

260 

1 

210 

3 

Table  3.10  Comparison  of  Sorties  Predicted,  No-Attacks,  All  Factors  Low 


Pull 

Model 

10%  Significance 
Reduced  Model 

5%  Significance 
Reduced  Model 

Minimum  MSE 
Reduced  Model 

Near  Min  MSE 
Reduced  Model 

Day 

Sorties 

Sorties 

Bias 

Sorties 

Bias 

Sorties 

Bias 

Sorties 

Bias 

260 

264 

4 

266 

5 

261 

1 

263 

3 

169 

188 

19 

197 

28 

177 

8 

183 

14 

201 

iratll 

4 

206 

6 

210 

10 

5 

187 

189 

2 

188 

186 

-1 

190 

3 
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180 
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3 

176 

1 

181 

1 

179 

-0 

6 
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179 

-1 
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-8 

7 
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175 
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173 

-1 
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-1 
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-2 

9 
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2 
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2 

163 

1 
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-3 

10 

147 

148 

1 

152 

5 

144 

-3 

148 

1 

11 

151 

147 

-4 

150 

-1 

151 

0 

147 

-4 

12 

145 

144 

-1 

145 

-1 

142 

-3 

144 

-1 

13 

140 

144 

4 
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-0 

140 

1 

145 

6 

14 
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131 

3 

131 

2 

131 

2 
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1 

15 
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-5 
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126 
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-7 

16 

127 

129 

1 
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-5 

17 
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112 
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3 
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A 
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114 

112 

-2 

19 

HE] 
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76 
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73 

H 

72 
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69 

-3 

66 

■3 

68 

MB 

Mean 

1  Error  1 

■ 
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n 

MAPE 

3.7% 

5.1% 

2.7% 

_JU%] 

3.10.2.2  Variance.  Tables  3.11  and  3.12  clearly  highlight  the 
reason  for  rejecting  the  full  model  for  either  estimation  or  prediction,  that 
is,  the  much  higher  variance  than  the  reduced  models.  As  explained  earlier, 
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the  hill  models  contain  many  terms  that  add  to  the  variance  without 
significantly  affecting  the  predictive  power  of  the  model.  The  choice 
between  the  reduced  models  is  not  as  clear  because  they  all  achieve  similar 
results  in  terms  of  overall  variance  for  prediction.  Variance  for  estimation, 
on  the  other  hand,  is  a  function  of  number  of  terms  in  the  model,  and  the 
10%  significance  models  always  achieve  the  best  result  measured  by 
estimation  variance,  although,  as  we  saw  in  Tables  3.7  through  3.10,  at  the 
expense  of  generally  higher  bias.  The  differences  in  variance  are  relatively 
small  when  we  consider  that  prediction  intervals  are  based  on  the  square 
root  of  variance,  so  the  three  sets  of  reduced  models  can  be  considered 
practically  equivalent  in  achieving  low  values  for  prediction  variance. 


The  models 


selected  by  choosing  terms  with  a  minimum  marginal  contribution  to  MSE 


(near  minimum  MSE)  were  successful  in  achieving  the  aim  of  the  lowest 
prediction  variance,  but  the  difference  between  their  variance  and  that  of 


the  other  models  is  not  as  great  as  was  hoped  for.  Such  a  selection 
technique  does  however  warrant  consideration,  because  it  strikes  a  balance 


between  models  selected  without  regard  to  variance  (stepwise  or  forward 
selection),  and  the  minimum  MSE/maximum  adjusted  R2  technique,  which 
does  not  consider  the  marginal  change  in  overall  variance  as  terms  are 
added.  The  modified  technique  should  be  generally  applicable  to  models 
where  there  are  many  possible  terms  to  be  included  and  the  ratio  of  MSE  to 
the  sum  of  the  predictor  variances  is  relatively  low.  For  this  data  set  the 
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ratio  is  typically  five  to  one,  which  implies  that  the  predictor  variances  are 
important  in  determining  the  overall  variance. 


Table  3.11  Prediction  Model  Variances,  Attack  Case 


Note:  Prod  »  variant*  of  a  new  prediction;  Eat  -  variance  of  Patimat*  of  expected  value  of  aortiea  down;  p  -  no 
of  tonne  in  modal,  fachnBng  intercept. 


na 

10%  Reduced  Model 
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Eat 
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P 
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61.1 
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43.7 
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9 
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883 
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464 
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28 

438 

94.1 

344 
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431 

846 

347 

22 
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386 

14 

876 

53.1 
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28 
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803 

296 

21 
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28 
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280 
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10 

434 
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28 

436 

89.7 
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422 
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14 
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62.4 
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12 
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37.1 
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21 
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14 
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37.7 

229 

27 

273 

57.4 
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10.2 
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ji 

101.1 
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Table  3.12  Prediction  Model  Variances,  No-Attack  Case 


Note:  Prod  »  variance  of  a  now  prediction;  Est  =  variance  of  estimate  of  expected  value  of  sorties  flown;  p  -  no 
of  terms  in  model,  including  intercept. 
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439 

144 

19 

182 
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10 
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22 
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39.7 
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18 
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34.7 
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12 
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29 
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14 
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27 
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29 
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12 
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27 
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23 
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362 

12 
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ns 
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31.7 
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E3 

3.1Q.4  Selection  of  Prediction  Models.  For  this  data  set,  when  the 
purpose  of  the  metamodel  is  prediction  the  preferred  models  are  those 
developed  using  minimum  MSE  (and  thus  maximum  adjusted  R3)  as  the 
criterion  for  selecting  variables.  The  consistently  low  bias  exhibited  by 
these  models  outweighs  the  marginally  higher  variances  compared  to  the 
other  models.  The  low  bias,  relative  to  the  full  models,  is  a  function  of 
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including  more  terms  than  the  other  models,  which  is  further  reason  to 
recommend  these  models,  given  that  testing  for  bias  was  accomplished  at 
only  two  design  points.  The  most  conservative  approach  is  to  include  as 
many  terms  as  possible,  without  excessively  inflating  variance  and  the 
minimum  MSE  models  achieve  this  goal  better  than  all  the  others. 

3.10.5  The  Prediction  Models.  The  models  chosen  for  prediction  are 
presented  in  Appendix  A  and  Appendix  B  for  the  attack  and  no-attack  cases 
respectively.  The  minimum  F  value  is  4.5  for  the  attack  models  is  and  3.8 
for  the  no-attack  models,  with  corresponding  probabilities  of  achieving  those 
values  of  F  by  chance  of  no  more  than  0.0001.  Many  of  the  terms  in  the 
models  would  be  considered  insignificant  in  an  explanatory  model,  with 
values  of  t  clore  to  1,  and  significance  levels  for  the  least  significant  variable 
typically  0.3,  but  we  have  seen  that  the  terms  are  important  in  controlling 
bias  without  contributing  to  excess  variance.  Finally,  residual  analysis 
revealed  nothing  to  suggest  that  the  residuals  are  other  than  normally 
distributed  with  expected  value  zero. 

3.10.6  Validation  of  the  Models.  To  better  validate  the  prediction 
models,  comparisons  should  be  made  between  the  predicted  values  and  the 
results  of  simulation  runs  carried  out  at  design  points  other  than  those  used 
in  the  original  simulation.  This  is  a  task  that  is  outside  the  scope  of  this 
research. 
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3.11  Models  for  Explanation 

3.11.1  Overview.  Developing  models  to  explain  the  behaviour  of  the 
►  airbase  system,  based  on  the  results  of  a  simulation  experiment,  is  a  much 

more  subjective  task  than  developing  models  for  prediction.  In  general,  we 
k  are  seeking  models  with  as  few  terms  as  will  adequately  identify  the  most 

important  factors  in  airbase  operability.  In  practical  terms,  it  is  important 
that  we  find  the  terms  that  have  the  greatest  effect  on  the  performance  of 
the  base  because  it  is  unlikely  that  there  will  ever  be  sufficient  resources  to 
optimise  the  level  of  all  the  factors.  It  is  also  important  that  we  know  how 
large  an  effect  that  the  chosen  factors  have  on  sortie  generation  so  that 
policy  makers  can  relate  the  cost  of  resource  allocations  to  the  additional 
level  of  capability  provided.  For  example,  the  factor  that  has  the  most 
impact  on  number  of  sorties  generated  may  be  prohibitively  expensive  to 
provide  at  its  high  level,  while  several  less  important  factors  may  provide 
worthwhile  gains  relative  to  their  cost.  The  form  of  regression  metamodel 
proposed  by  Diener  (1989:42-43)  effectively  provides  the  information 
required.  The  magnitudes  of  the  regression  coefficients  indicate  both  the 
effect  that  a  resource  has  on  sortie  generation,  and,  when  compared  to  the 
other  coefficients,  the  relative  importance  of  that  factor.  Relative 
importance  can  also  be  assessed  by  examining  the  contribution  that  a  factor 
makes  to  the  explanation  of  variance  in  a  model,  that  is,  the  partial  R3  for 
that  factor.  Study  of  the  partial  R3  is  useful  because  the  regression 
coefficients  tend  to  understate  the  differences  in  relative  importance 


3.59 


between  the  factors.  For  example,  the  most  important  terms  commonly 
have  a  regression  coefficient  less  than  double  that  of  the  next  most 
important  term,  but  contribute  three  or  more  times  as  much  to  the 
explanation  of  variance  as  measured  by  partial  R3. 


3.12  Using  Significance  Levels  to  Develop  Explanatory  Models 

One  approach  to  developing  explanatory  models  is  to  presuppose  a 
minimum  significance  level  for  the  variables  to  be  included  in  a  model,  and 
then  use  some  form  of  stepwise  regression  technique  to  find  the  model  that 
meets  the  significance  criterion.  We  have  already  seen  that  because  the 
variables  in  this  data  set  have  completely  independent  effects  on  a  model, 
forward  selection  is  the  most  efficient  technique,  and  is  certain  to  find  the 
'best'  subset  of  variables  which  are  significant  at  the  specified  level,  where 
'best'  implies  the  subset  with  the  highest  R3  value.  A  corollary  to  the 
previous  observation  is  that  models  developed  using  a  low  significance  level 
(a)  will  always  be  subsets  of  models  with  higher  significance  levels,  thus 
implying  that  a  variable  that  is  not  included  in  a  model  at,  for  example,  the 
0.10  level  will  never  appear  in  models  with  terms  significant  at  less  than 
a  =  0.10. 


3.12.1  Forward  Selection.  By  setting  a  significance  level  at  which  a 
term  is  able  to  enter  a  model,  we  are  testing  the  null  hypothesis  H<,  that  the 
true  value  of  the  coefficient  for  that  term  is  zero.  The  chosen  significance 
level  a  represents  the  probability  of  Type  I  error  in  the  test,  that  is,  the 
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probability  of  including  a  term  in  the  model  when  in  fact  it  has  no  effect. 

For  our  purpose  of  identifying  the  important  factors  affecting  sortie 
generation,  a  relatively  low  significance  level  would  appear  to  be 
appropriate  so  that  we  can  concentrate  on  factors  that  are  highly  unlikely  to 
be  in  a  model  by  chance  alone.  We  already  have  models  that  were 
developed  with  a  significance  level  of  0.10,  so  appropriate  levels  of  a  for 
further  investigation  are  0.05  and  0.01.  Lower  significance  levels  than  used 
for  the  existing  models  are  chosen  for  two  reasons.  First,  as  we  observed  in 
Chapter  I,  the  existing  models  contain  so  many  terms  that  analysis  and 
explanation  are  difficult;  and  second,  lower  significance  levels  reduce  the 
risk  of  rejecting  true  null  hypotheses  when  many  tests  are  carried  out.  At 
lower  significance  levels  fewer  terms  will  be  included  and  hopefully  a 
clearer  relationship  between  the  variables  and  sorties  generated  will 
emerge.  The  compromise  we  make,  however,  is  that  our  measure  of  fit,  R2, 
reduces  as  variables  are  removed.  To  illustrate  the  effect  of  reducing  the 
significance  level,  Tables  3.13,  3.14,  and  3.15  present  the  main  effects 
included  in  the  metamodels  for  the  attack  case  with  terms  significant  at  the 
0.10,  0.05,  and  0.01  levels  respectively.  The  complete  metamodels  are 
presented  in  Appendix  C,  Tables  C.l,  C.2,  and  C.3.  Tables  D.l,  D.2,  and 
D.3  in  Appendix  D  contain  complete  metamodels  for  the  no-attack  case  at 
the  same  three  significance  levels  used  in  the  attack  case.  Tables  3.16  and 
3.17  compare  the  number  of  terms  included  and  the  R2  value  for  each  model 
for  the  attack  case  and  no-attack  case  respectively.  The  unadjusted  R2  is 


tabulated  to  show  how  much  of  the  total  explainable  variance  a  reduced 
model  captures,  with  the  full  model  showing  the  man'mum  that  can  be 
explained  for  any  day. 


Table  3.13  Daily  Metamodels,  Attack  Case,  Alpha  =  0.10 


Main 

DAY  Intercept  Attrit 
A 


89.6 

83.2 
104.3 

92.3 

5  102.1 

6  166.2 


7  148.6 

8  145.7 

9  137.3 

10  132.3 

11  127.1 

12  121.2 


13 

14 

15  106.8 

16  100.0 

17  95.9 

18  89.0 


19  86.5 

20  79.2 

21  74.5 

22  69.6 

23  65.7 

24  62.1 


59.0 

55.6 

51.6 

48.7 
46.6 
42.9 


sets 

Fill  ABDR  Recov  Pers 

B  C  D  E 


12.0 

40.6 

22.3  3.8 
-3.1  9.5 

8.7  6.9  4.1 

10.8  5.4 


AIS  Spt  Eq  Spares  Miss  Fuel 
F  G  H  J  K 


L6.8 

9.7 

L4.1 

8.5 

5.3 

L5.6 

4.3 

7.8 

4.6 

L3.4 

3.4 

7.7 

4.7 

L2.7 

6.8 

4.6 

L1.2 

7.3 

5.2  3.7 

6.0 

7.1 

6.1 
7.7 
6.2 

2.8  5.5 


4.5 

5.5 

2.5  5.6 

2.6  3.9 

3.5 


5.1  4.1 
3.6 

5.8 

4.8 
4.8 

5.2 


5.1 

4.8 

3.8 

4.8 
3.5 

4.2 


3.62 


Table  3.14  Daily  Metamodels,  Attack  Case,  Alpha  =  0.05 


Main  Effects 

DAY 

Intercepi 

Attrit 

Fill  ABDR 

Recov 

Pers 

AIS 

SptEq 

Spares 

Miss 

Fuel 

A 

B  C 

D 

E 

F 

G 

H 

J 

K 

89.6 

12.0 

83.2 

40.6 

104.3 

22.3 

92.3 

9.5 

5 

102.1 

8.7 

6.9 

4.8 

6 

166.2 

10.8 

11.3 

• 

7 

148.6 

10.6 

9.0 

8 

145.7 

15.3 

5.9 

9 

137.3 

12.2 

6.8 

9.1 

10 

132.3 

14.2 

10.1 

11 

127.1 

15.8 

9.7 

12 

121.2 

16.0 

8.9 

5.3 

13 

116.5 

16.8 

9.7 

14 

14.1 

8.5 

5.3 

15 

15.6 

7.8 

4.6 

16 

13.4 

7.7 

4.7 

17 

95.9 

12.7 

6.8 

4.6 

18 

11.2 

7.3 

5.2 

19 

86.5 

12.4 

6.0 

5.1 

4.1 

20 

79.2 

11.3 

7.1 

21 

74.5 

10.3  3.6 

6.1 

5.8 

22 

69.6 

8.9 

7.7 

4.8 

23 

65.7 

9.6 

6.2 

4.8 

24 

62.1 

10.4 

5.5 

5.2 

25 

9.9 

4.5 

5.1 

26 

55.6 

8.8 

5.5 

4.8 

27 

51.6 

9.6 

5.6 

4.7 

28 

48.7 

8.6 

3.9 

4.8 

29 

46.6 

7.8 

3.5 

3.5 

30 

42.9 

7.6 

3.3 

4.2 

3.63 


Table  3.15  Daily  Metamodels,  Attack  Case,  Alpha  =  0.01 


Main  Effects 

DAY 

Intercep 

Attrit 

Fill  ABDR 

Recov 

Pers 

A1S 

SptEq 

Spares 

Miss 

Fuel 

A 

B  C 

D 

E 

F 

G 

H 

J 

K 

89.6 

12.0 

83.2 

40.6 

104.3 

22.3 

92.3 

9.5 

5 

102.1 

8.7 

6.9 

6 

166.2 

10.8 

11.3 

7 

148.6 

10.6 

9.0 

8 

145.7 

15.3 

9 

137.3 

12.2 

6.8 

9.1 

10 

132.3 

14.2 

10.1 

11 

127.1 

15.8 

9.7 

12 

121.2 

16.0 

8.9 

13 

116.5 

16.8 

9.7 

14 

14.1 

8.5 

15 

15.6 

7.8 

16 

13.4 

7.7 

17 

95.9 

12.7 

6.8 

18 

11.2 

n 

19 

86.5 

12.4 

6.0 

20 

79.2 

11.3 

7.1 

21 

74.5 

10.3 

6.1 

5.8 

22 

69.6 

8.9 

7.7 

4.8 

23 

65.7 

9.6 

6.2 

24 

62.1 

10.4 

5.5 

5.2 

25 

59.0 

9.9 

5.1 

26 

55.6 

8.8 

5.5 

4.8 

27 

51.6 

9.6 

4.7 

28 

48.7 

8.6 

4.8 

29 

46.6 

7.8 

30 

42.9 

7.6 

4.2 

3.12.2  Assessment  of  Models.  Attack  Case.  The  models  developed 
using  a  significance  level  of  5%  show  the  important  variables  and  their 
interactions  more  clearly  than  the  baseline  10%  models.  Comparison  of 
Table  C.l  with  Table  C.2  shows  that  in  general,  the  terms  that  the  5% 
models  exdude  appear  only  in  isolated  instances  in  the  10%  models. 
Although  a  few  terms  still  appear  on  only  one  or  two  days,  there  are  far 
fewer  isolated  occurrences  of  variables.  The  more  consistent  inclusion  of 
variables  makes  the  determination  of  the  important  factors  and  their 
interactions  easier.  The  5%  models  clearly  show  that  there  are  relatively 
few  important  factors  and  interactions,  and  that  most  of  the  significant 
interactions  only  have  effects  in  the  second  half  of  the  thirty-day  period. 

The  notable  exception  is  the  interaction  between  spares  and  filler  aircraft, 
which  shows  a  strong  reinforcing  effect  during  the  six  day  attack  period. 
Table  C.3  shows  that  the  models  with  terms  significant  at  1%  exdude  terms 
that  appear  consistently  in  the  5%  and  10%  models.  For  example,  all  the 
missiles  and  fuel  and  spares  and  fuel  interactions,  and  most  of  the 
personnel  and  missiles  and  AIS  and  fuel  interactions  are  exduded. 

Although  we  can  be  more  certain  at  a  significance  level  of  1%  that  the  terms 
induded  in  a  model  are  truly  significant,  the  risk  of  Type  II  error,  that  is, 
the  risk  of  rejecting  terms  that  should  be  included,  is  higher  than  for 
a  =  0.05  or  a  =  0.10.  Although  no  measure  of  the  Type  II  error  was 
calculated,  the  wholesale  elimination  in  the  1%  models  of  terms  consistently 
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important  in  both  the  larger  models  suggests  that  the  risk  of  Type  II  error 
is  unacceptably  high. 


Table  3.16  Comparison  of  Ra  Values  for  Attack  Case  Models 


Day 

R*  Pull 
Model 

RJ  10% 
Significance 

Terms 

R*  5% 
Significance 

Terms 

R*  1% 
Significance 

Terms 

0.75 

0.71 

2 

0.71 

0.71 

1 

0.94 

0.93 

8 

0.92 

0.92 

3 

0.82 

0.78 

5 

0.77 

0.76 

2 

0.72 

0.61 

10 

0.54 

0.52 

2 

5 

0.70 

0.62 

10 

0.57 

0.52 

2 

6 

0.59 

0.48 

11 

0.37 

0.34 

2 

0.69 

0.53 

8 

0.48 

0.46 

2 

0.62 

0.47 

8 

0.41 

0.35 

1 

0.66 

0.53 

8 

0.48 

0.48 

3 

10 

0.65 

0.51 

7 

0.48 

5 

0.44 

2 

11 

0.71 

0.60 

11 

0.54 

6 

0.48 

2 

12 

0.73 

0.63 

10 

0.59 

n 

0.52 

2 

13 

0.71 

0.61 

7 

0.58 

wM 

0.56 

2 

14 

0.71 

0.59 

8 

0.55 

5 

0.51 

2 

16 

0.68 

0.61 

10 

0.56 

6 

0.52 

3 

16 

0.72 

0.62 

11 

0.59 

8 

0.50 

2 

17 

0.70 

0.60 

11 

0.57 

9 

0.46 

2 

18 

0.69 

0.57 

10 

0.56 

9 

0.42 

2 

19 

0.69 

0.56 

9 

0.56 

9 

0.43 

2 

0.72 

0.63 

13 

0.56 

9 

0.44 

2 

21 

0.74 

0.65 

14 

0.64 

13 

0.51 

4 

22 

0.74 

0.64 

14 

0.58 

9 

0.48 

3 

23 

0.69 

0.56 

10 

0.52 

0.44 

2 

24 

0.74 

0.65 

14 

0.60 

0.50 

3 

25 

0.74 

0.62 

10 

0.62 

10 

0.48 

3 

26 

0.71 

0.62 

12 

0.60 

10 

0.54 

6 

27 

0.77 

0.70 

15 

0.67 

11 

0.57 

5 

28 

0.72 

0.64 

12 

0.62 

10 

0.52 

4 

29 

0.71 

0.62 

14 

0.58 

11 

0.43 

2 

0.71 

0.63 

13 

0.59 

9 

0.50 

4 

|  Mean 

0.72 

0.62 

10.2 

0.58 

7.1 

0.51 

2.5 

3.12.3  Comparison  of  Ra  Values.  Attack  Case.  Table  3.16  shows  that 
for  all  but  a  few  days,  the  5%  models  achieve  Ra  values  nearly  as  high  as 
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the  larger  10%  models.  The  5%  models  average  81%  of  the  full  model  Ra, 
compared  to  86%  for  the  10%  models  and  71%  for  the  1%  models.  The  table 
also  shows  during  the  first  fifteen  days  the  1%  models  also  achieve  R3 
values  closely  comparable  with  the  10%  models,  but  in  the  last  fifteen  days, 
the  performance  of  the  1%  models  declines  markedly.  The  lower  R3  values 
for  the  1%  models  in  the  last  half  of  the  period  correspond  well  with  the 
rejection  of  most  of  the  interaction  terms  significant  in  the  larger  models. 

3.12.4  Recommended  Models.  Attack  Case.  For  the  attack  case,  the 
models  with  terms  significant  at  a  level  of  0.05  are  considered  to  be  the  best 
compromise  for  identifying  the  most  important  factors  and  interactions, 
while  still  capturing  most  of  the  explainable  variance  in  the  models.  The 
minimum  value  of  F  for  any  of  the  5%  models  is  6.8,  with  a  corresponding 
probability  of  F  or  higher  of  no  more  than  0.0001.  Analysis  of  the  residuals 
for  the  5%  models  included  normal  probability  plots,  and  plots  of  residuals 
against  fitted  values.  With  the  exception  of  the  plot  of  residuals  against 
fitted  values  for  the  first  day,  no  indications  were  found  to  suggest  that  the 
models  are  invalid.  The  unusual  residual  plot  for  the  first  day  is  attributed 
to  the  inclusion  of  only  one  variable,  besides  the  blocking  terms,  in  the 
model  for  that  day.  The  plot  is  included  in  Appendix  J. 

3.12.5  Assessment  of  Models.  No-Attack  Case.  The  assessment  of  the 
no-attack  case  models  is  not  as  straightforward  as  for  the  attack  case 
models.  Table  D.l  shows  that  the  generally  consistent  appearance  of  factors 
over  time  observed  in  the  attack  case  models  is  less  pronounced  in  the 
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Table  3.17  Comparison  of  Ra  Values  for  No- Attack  Case  Models 


Day 

R2  Full 
Model 

R2  10% 
Significance 

Terms 

R2  5% 
Significance 

Terms 

R2  1% 
Significance 

Terms 

1 

0.65 

0.53 

0.49 

0.47 

3 

2 

0.68 

0.57 

0.52 

0.48 

2 

3 

0.70 

0.61 

0.50 

0.48 

2 

4 

0.61 

0.45 

0.42 

0.40 

3 

S 

0.64 

0.50 

0.46 

0.38 

2 

6 

0.79 

0.69 

0.65 

0.57 

6 

7 

0.64 

0.45 

0.43 

0.38 

2  | 

8 

0.64 

0.47 

0.36 

3 

9 

0.82 

0.71 

0.69 

0.67 

4 

10 

0.82 

0.77 

0.74 

0.71 

6  I 

11 

0.79 

0.73 

0.73 

0.69 

3 

12 

0.81 

0.74 

0.67 

0.66 

2  1 

13 

0.85 

0.78 

0.75 

0.72 

3  I 

14 

0.76 

0.68 

0.66 

0.65 

3  1 

15 

0.80 

0.74 

0.70 

0.66 

2 

16 

0.84 

0.78 

0.76 

0.69 

2 

17 

0.87 

0.80 

0.76 

0.73 

2 

18 

0.86 

0.81 

0.78 

0.69 

3 

19 

0.74 

0.66 

0.60 

0.44 

5 

20 

0.78 

0.70 

0.68 

0.50 

5 

21 

0.84 

0.77 

0.71 

0.65 

3 

22 

0.80 

0.72 

0.70 

0.65 

4 

23 

0.83 

0.74 

0.71 

0.64 

2 

24 

0.82 

0.73 

0.66 

0.60 

2 

25 

0.73 

0.62 

0.60 

0.57 

5 

26 

0.70 

0.57 

0.52 

0.45 

4 

27 

0.64 

0.50 

0.36 

0.30 

4 

28 

0.71 

0.60 

0.55 

0.41 

3  1 

29 

0.76 

0.67 

0.61 

0.56 

4 

30 

0.75 

0.67 

0.66 

0.55 

4 

Mean 

0.76 

0.66 

0.62 

0.56 

3.3 

no-attack  models,  with  terms  in  the  10%  models  often  included  for  a  day  or 
two,  and  then  reappearing  several  days  later.  Determining  what  effects  are 
important  is  less  dear  cut.  For  example,  is  a  factor  that  appears 
sporadically  on  six  or  seven  occasions  important  compared  to  a  factor  that 
appears  on  five  consecutive  days?  Reducing  the  significance  level  to  5% 
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tends  to  remove  more  of  the  terms  that  occur  sporadically,  while  retaining 
intact  most  of  the  consecutive  groups.  Though  not  as  clear  as  the  5% 

„  models  in  the  attack  case,  the  5%  models  in  the  no-attack  case  are  easier  to 

analyse  than  the  10%  models  and  do  not  mask  any  factors  or  interactions 
-  that  are  apparently  important.  Also,  the  5%  models  do  not  seriously  distort 

when  the  factors  are  important  during  the  thirty  days.  The  1%  models, 
however,  remove  so  many  terms  that  if  not  for  the  magnitude  of  the 
interaction  coefficients  included,  particularly  for  the  fillers  and  fuel 
interaction  during  the  last  four  days,  we  could  conclude  that  interactions 
are  not  important  in  the  no-attack  case.  If,  using  the  1%  models,  we  decide 
that  a  factor  is  important  if  it  appears  on  several  consecutive  days,  we 
would  reach  conclusions  as  to  the  important  factors  similar  to  conclusions 
based  on  the  larger  models,  with  the  exception  of  the  fillers  and  fuel 
interaction.  We  would  not,  however,  notice  to  the  same  extent  that  some 
main  factors  and  interactions  are  more  or  less  important  depending  upon 
when  during  the  thirty  days  we  observe  their  effect. 

3.12.6  Comparison  of  Ra  Values.  No- Attack  Case.  The  5%  models 
achieve  on  average  82%  of  the  full  model  Ra,  compared  to  87%  for  the  10% 
models.  Reducing  the  significance  level  to  1%  results  in  a  more  substantial 
*  drop  in  R2,  with  the  1%  models  averaging  only  74%  of  the  full  model  R2. 

The  relatively  large  drop  in  R2  for  the  1%  models  suggests  that  they  do  not 
‘  fit  the  data  particularly  well. 
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A  significance  level  of 


a  12.7  Recommended  Models.  No-Attack  Case, 

5%  again  results  in  the  best  compromise  between  model  fit  and  ease  of 

interpretation.  The  5%  models  are  all  highly  significant,  with  a  minimum  F  , 

value  of  4.9,  and  all  p- values  no  more  than  0.0001.  Analysis  of  the 

residuals  shows  no  serious  departures  from  normality,  and  plots  of  the 

residuals  against  the  fitted  values  also  indicate  no  serious  shortcomings  in 

the  models,  except  possibly  for  the  last  three  days.  Some  of  the  residuals 

fall  along  a  straight  line  angled  through  the  origin,  although  the  majority 

are  randomly  scattered.  The  straight  line  marks  a  boundary,  beyond  which 

no  residuals  are  observed.  The  reason  for  the  boundary,  which  also  appears 

in  the  residual  plots  for  the  10%  models,  was  not  established.  The  plots  for 

the  last  three  days  for  both  significance  levels  are  included  in  Appendix  J. 

3.12.8  Statistical  Comparisons  of  Models.  Both  Cases.  We  have 
already  noted  that  models  with  lower  significance  levels  are  subsets  of 
models  with  higher  significance  levels,  so  it  is  possible  to  test  the  smaller 
models  against  the  larger  using  a  general  linear  test  (GLT),  with  the  aim  of 
determining  whether  or  not  the  terms  not  in  the  smaller  model  are 
significantly  different  from  zero.  For  the  purposes  of  the  test,  the  larger 
model  is  equivalent  to  a  full  model,  and  the  smaller  model  is  the  reduced  or 
constrained  model.  Setting  a  null  hypothesis  that  the  coefficients  of  the  « 

terms  not  included  in  the  smaller  model  were  equal  to  zero  and  testing  the 
5%  significance  models  against  the  10%  models  resulted  in  much  higher  ‘ 

than  expected  values  of  F' .  Because  both  numerator  and  denominator 
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degrees  of  freedom  are  different  in  most  models,  p-values  were  calculated, 
resulting  in  a  maximum  p-value  for  the  F  statistic  of  0.09  for  both  the 
attack  and  no-attack  case,  and  an  average  value  of  0.02  over  the  thirty 
models  in  each  case.  Such  low  p-values  would  lead  to  rejection  of  the  null 
hypothesis  with  a  =  0.05  on  all  but  three  days  in  the  attack  case,  and  all 
but  four  days  in  the  no-attack  case.  Although  an  initial  conclusion  might  be 
that  the  5%  models  are  inadequate,  further  examination  of  the  variables 
being  excluded  and  the  nature  of  the  GLT  reveals  that  the  low  p-values  are 
what  we  would  expect  to  see,  and  that  the  GLT  is  of  relatively  little  use  in 
comparing  the  models. 

3.12.9  Limitations  of  the  General  Linear  Test.  The  first  point  to  note 
is  that  the  p-value  exceeds  0.05  only  when  the  larger  and  smaller  models 
differ  by  only  one  variable.  Also,  for  those  seven  days,  the  significance  level 
of  the  last  term  in  the  larger  model  is  equal  to  the  calculated  p-value  for  the 
GLT.  The  GLT  therefore  is  able  to  accurately  measure  the  effect  of  a  single 
variable,  but  when  the  difference  between  models  is  more  than  one  variable, 
we  are  testing  the  combined  effect  of  all  the  variables  not  included  in  the 
smaller  model.  It  is  also  important  to  note  that  the  variables  being  tested 
by  the  GLT  have  similar  sums  of  squares,  and  that  because  they  were 
included  in  the  larger  model,  must  have  been  significant  at  between  10% 
and  approximately  5%.  If  their  significance  level  had  been  appreciably  less 
than  5%  in  the  larger  model,  they  would  have  been  included  in  the  smaller 
model,  and  would  not  be  the  subject  of  our  test.  The  GLT  considers  the 
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relative  difference  in  SSE  between  the  models,  and  the  change  in  degrees  of 
freedom,  so  that  two  or  more  variables,  which  must  have  substantial  sums 
of  squares  to  have  been  in  the  larger  model  in  the  first  place,  will  have  a 
large  effect  on  SSE,  for  a  very  small  change  in  degrees  of  freedom  compared 
to  the  denominator  degrees  of  freedom.  For  example,  on  day  ten  in  the 
attack  case,  the  last  two  variables  that  appear  in  the  10%  model  are 
excluded  from  the  5%  model.  Their  significance  levels  in  the  10%  model  are 
0.0613  and  0.0668,  and  their  sums  of  squares  are  2664  and  2556.  The  sum 
of  squares  of  the  last  variable  included  in  the  5%  model  is  3301,  so  the 
combined  effect  of  the  two  excluded  variables  relative  to  variables  that  are 
included  is  sufficient  for  the  GLT  to  return  a  low  p-value  of  0.03.  In 
hindsight,  the  GLT  is  inappropriate  to  compare  models  developed  using 
significance  levels.  For  a  single  variable  not  included  in  a  smaller  model, 
we  already  know  that  it  is  not  significant  at  a  given  level,  or  it  would  have 
been  included.  For  groups  of  variables,  we  already  know  that  they  are  all 
individually  not  significant  in  the  smaller  model,  and  testing  as  a  group 
cannot  add  additional  information,  and  in  fact  may  be  misleading  because  of 
the  combined  effects  of  several  moderately  significant  variables. 

3.13  An  Alternate  Approach  to  Model  Development 

An  alternate  approach  to  developing  explanatory  models  is  to  take 
advantage  of  the  independence  of  the  variables  and  examine  their  effect  and 
importance  individually.  Because  the  variables  act  independently  we  know 
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that  the  most  significant  variable  in  a  full  model  will  always  be  the  most 
significant  in  a  reduced  model,  and  similarly  for  the  second  most  significant 
and  subsequent  variables.  To  illustrate,  if  we  wish  to  examine  a  model  with 
two  terms,  we  simply  rank  the  terms  that  appear  in  a  full  model  by  their 
sum  of  squares  contribution,  and  choose  the  first  two,  knowing  that  their 
significances  will  not  reduce  when  combined,  and  conversely,  that  none  of 
the  53  terms  excluded  could  increase  in  significance  if  combined  either  with 
the  selected  terms  or  each  other.  As  an  approach  to  model  reduction, 
knowing  the  order  in  which  terms  will  enter  a  model  and  the  contribution 
that  they  will  make  allows  all  model  sizes  to  be  very  quickly  evaluated 
without  having  to  carry  out  any  computer  runs  except  an  initial  full  model 
regression.  A  significant  advantage  to  evaluating  all  the  variables  based  on 
their  contribution  to  a  model  is  that  any  distinct  groups  of  variables  making 
similar  contributions  can  be  readily  identified,  and  either  included  or 
excluded  as  a  group,  instead  of  being  possibly  split  by  a  technique  based  on 
significance  level. 

3.13.1  Outline  of  Technique.  Before  outlining  this  alternative  to 
stepwise  regression  we  stress  that  this  technique  is  only  applicable  when 
the  variables  make  independent  contributions  to  a  regression  model,  that  is 
the  Type  I  and  Type  in  sums  of  squares  are  equal  for  all  variables.  A  key 
point  for  the  use  of  this  procedure  is  to  recognise  that  of  the  thousands  of 
variable  combinations  possible,  only  55  practically  useful  models  exist, 
because  it  is  counterproductive  in  terms  of  R3  to  add  variables  to  a  model  in 
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other  than  their  sorted  sequence.  For  example,  a  six  variable  model 
containing  the  first  five  variables  and  the  seventh  will  not  fit  as  well  as  the 
model  containing  the  first  six  variables.  If  useful  models  must  contain 
terms  in  their  sorted  sequence,  then  the  number  of  models  to  consider 
reduces  to  the  total  number  of  available  terms,  that  is  55.  The  starting 
point  for  this  technique  is  to  produce  a  full  model  to  obtain  analysis  of 
variance  information,  sums  of  squares  for  each  variable,  an  intercept 
parameter  estimate,  and  parameter  estimates  for  all  the  variables.  The 
SAS  REG  procedure  is  capable  of  producing  all  the  necessary  output.  The 
next  step  is  to  sort  the  output  in  decreasing  order  of  the  sum  of  squares  for 
each  variable.  This  sorted  output  now  represents  the  order  in  which  the 
variables  would  be  selected  by  an  automatic  forward  selection  procedure.  A 
model  therefore  comprises  the  last  term  considered,  and  all  the  previous 
terms  in  the  sorted  list.  Given  that  the  total  sum  of  squares  for  any  model 
(SSTO)  is  a  constant,  it  is  a  simple  matter  to  calculate  partial  R3  for  every 
variable.  The  cumulative  total  of  the  partial  sum  of  squares,  starting  with 
the  most  significant  variable  and  ending  with  the  last  variable  for  that 
model  size  is  thus  the  model  R2.  Adjusted  R3,  Cp,  and  MSE  for  each  model 
size  may  be  readily  calculated,  as  well  as  the  value  of  the  t  statistic  for  the 
last,  that  is,  least  significant  variable  to  enter  the  model.  Recalling  that  we 
found  the  variances  for  each  predictor  to  be  equal  in  any  model,  the  common 
value  for  the  variance  of  the  predictors  in  each  size  model  may  also  be 
calculated.  An  example  of  the  tabulated  data  for  the  first  twenty  models  for 
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the  first  day  of  the  no-attack  case  is  provided  in  Table  E.l  in  Appendix  E. 
Note  that  in  Table  E.l,  the  Type  I  and  Type  II  sums  of  squares  for  the 
blocking  terms  are  not  equal.  This  does  not  affect  the  analysis,  because  the 
blocking  terms  are  always  the  first  seven  terms  forced  into  any  model. 

3.13.2  Application  of  the  Technique.  Once  the  variables  are  sorted 
and  their  contributions  to  the  model  tabulated,  the  relative  importance  of 
each  variable  is  quickly  apparent.  Decision  rules  for  selection  of  variables 
may  then  be  applied.  For  example,  a  simplistic  decision  rule  could  be  to 
include  only  the  first  five  variables  in  a  model,  regardless  of  significance. 
Another  rule  could  be  to  only  include  variables  explaining  at  least  one 
percent  of  the  overall  variance.  Any  number  of  decision  rules  can  be 
formulated:  the  point  is  that  the  modeler  has  some  additional  flexibility  in 
specifying  the  model,  and  better  knowledge  of  how  the  variables  relate  to 
that  model.  An  additional  application  is  that  models  produced  using 
stepwise  methods  can  be  quickly  compared  to  the  sorted  list  to  ensure  that 
no  overly  arbitrary  divisions  among  the  variables  have  occurred. 

3.13.3  Limitations.  The  technique  described  in  the  preceding 
paragraphs  is  specifically  intended  to  assist  in  the  process  of  deciding  which 
variables  to  include  in  a  model  and  does  not  provide  all  the  information  of  a 
computer  generated  regression  model,  particularly  F  and  t  values  and  their 
associated  probabilities.  Once  the  variables  have  been  selected,  however, 
running  a  computer  regression  with  those  variables  selected  will  provide  the 
required  details. 


3.13.4  Decision  Rule.  The  technique  described  above  was  used  to 


choose  variables  which  were  then  forced  into  regression  models.  The 
decision  rule  used  to  choose  variables  involved  some  degree  of  subjectivity, 
because  the  full  models  are  all  very  different.  The  basic  rule  was  to  look  for 
a  reasonably  dear  division  between  the  most  significant  variables  and  the 
less  significant  remainder,  but  some  modifications  were  required.  Where 
one  variable  was  dominant  in  the  model,  the  next  most  important  variable 
(or  group  of  variables)  was  also  induded  if  it  was  distinct  from  the 
remainder.  Where  no  particularly  dear  break  was  evident,  a  cutoff  of 
approximately  one  percent  difference  between  successive  partial  R2  values 
was  used.  In  some  cases,  the  dedsion  rule  caused  a  substantial  reduction  in 
R2,  but  was  dearly  able  to  identify  the  most  important  variables.  The 
resulting  model  for  the  attack  case  are  presented  in  Table  F.l  in  Appendix 
F.  Table  G.l  in  Appendix  G  contains  the  models  for  the  no-attack  case. 

Comparisons  of  R2  for  these  models,  the  full  models  (maximum  possible  R2), 
and  the  1%  significance  models  are  induded  in  Table  3.18. 

3.13.5  Graphical  Summary  of  Potential  Models.  A  graphical 
summary  of  the  sorted  variables  is  induded  in  Appendices  H  and  I.  The 
graphs  contain  the  twenty  most  significant  variables  for  each  day,  and  show 
both  partial  R2  and  the  regression  coeffident  for  each  variable.  As  the  « 

largest  model  developed  using  significance  levels  contains  seventeen 
variables,  twenty  variables  were  induded  to  ensure  that  all  useful  model  * 
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Table  3.18  Comparison  of  R2  Values,  Subjective  Selection 


Day 

Ra,  Attack  Case 

R2,  No-Attack  Case  | 

Full 

Model 

1% 

Significance 

Subjective 

Selection 

Full 

Model 

1% 

Significance 

Subjective 

Selection 

0.75 

0.71 

0.71 

0.66 

0.47 

0.47 

0.94 

0.92 

0.92 

0.68 

0.48 

0.44 

0.82 

0.76 

0.77 

0.70 

0.48 

0.48 

0.72 

0.52 

0.54 

0.61 

0.40 

0.40 

5 

0.70 

0.52 

0.57 

0.64 

0.38 

0.41 

6 

0.59 

0.34 

0.41 

0.79 

0.57 

0.51 

7 

0.69 

0.46 

0.48 

0.64 

0.38 

0.43  ! 

8 

0.62 

0.35 

0.41 

0.64 

0.36 

0.36  | 

9 

0.66 

0.48 

0.48 

0.82 

0.67 

0.67 

10 

0.65 

0.44 

0.48 

0.82 

0.71 

0.64 

11 

0.71 

0.48 

0.48 

0.79 

0.69 

0.69 

12 

0.73 

0.52 

0.52 

0.81 

0.66 

0.63 

13 

0.71 

0.56 

0.56 

0.85 

0.72 

0.66 

14 

0.71 

0.51 

0.51 

0.76 

0.65 

0.62 

15 

0.68 

0.52 

0.49 

0.80 

0.66 

0.66 

16 

0.72 

0.50 

0.84 

0.69 

0.67 

17 

0.70 

0.45 

0.45 

0.87 

0.73 

0.73 

18 

0.69 

0.42 

0.42 

0.86 

0.69 

0.64 

19 

0.69 

0.43 

0.43 

0.74 

0.44 

0.44 

lei'll 

0.72 

0.44 

0.44 

0.78 

0.50 

0.50 

121 

0.74 

0.51 

0.51 

0.84 

0.65 

0.63 

0.74 

0.48 

0.45 

0.80 

0.66 

0.62 

0.69 

0.44 

0.47 

0.83 

0.64 

0.64 

24 

0.74 

0.50 

0.50 

0.82 

0.60 

0.60 

25 

0.74 

0.48 

0.48 

0.73 

0.57 

0.57 

26 

0.71 

0.54 

0.41 

0.70 

0.46 

0.50 

27 

0.77 

0.57 

0.57 

0.64 

0.30 

0.36 

28 

0.72 

0.52 

0.52 

0.71 

0.41 

0.38 

29 

0.71 

0.43 

0.43 

0.76 

0.56 

0.58 

30 

0.71 

0.50 

0.50 

0.75 

0.55 

0.55 

sizes  can  be  estimated.  Models  may  be  estimated  from  the  graphs  by 
including  the  desired  number  of  variables,  starting  with  the  variable  closest 
to  the  origin  and  progressively  adding  variables.  The  regression  coefficients 
can  be  read  from  the  graphs  because  we  showed  earlier  that  each  variable 
has  the  same  coefficient  in  a  model  regardless  of  the  other  terms  included. 
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The  intercept  values,  which  are  always  the  mean  of  the  observed  values, 
are  presented  in  Table  H.l  and  Table  1.1,  at  the  end  of  Appendix  H  and 
Appendix  I  respectively.  The  graphs  contain  the  essential  information 
presented  in  a  sorted  list  of  the  variables,  and  allow  an  immediate 
assessment  of  which  variables  are  important  on  a  given  day,  and  how  much 
they  affect  the  sorties  generated.  Although  partial  Ra  is  shown  for  each 
variable,  model  R3  cannot  be  estimated  from  the  graphs  because  the  effect  of 
the  blocking  variables  is  not  included.  The  regression  coefficients  and 
partial  Ra  share  a  common  scale,  where  regression  coefficients  are  in  units, 
and  partial  R3  is  expressed  as  a  percentage. 

3.13.6  Model  Assessment.  It  is  clear  from  the  tables  that  the  models 
developed  using  a  subjective  assessment  of  the  sorted  variables  are 
practically  identical  to  the  models  developed  using  a  significance  level  of 
1%,  with  respect  to  both  the  terms  appearing  in  the  models,  and  the  R3 
values  for  the  models.  Such  a  result  is  not  unexpected,  because  the  decision 
rule  chooses  the  few  variables  that  are  clearly  more  important  than  the 
remainder.  The  decision  rule  ensures  that  we  do  not  make  any  artificial 
divisions  between  variables  of  practically  equal  importance,  but  in  doing  so 
it  is  somewhat  biased  towards  choosing  a  small  number  of  variables  with 
high  relative  importance,  without  considering  that  some  of  the  less 
relatively  important  variables  may  well  be  important  in  the  absolute  sense. 
We  have  already  determined  that  the  1%  significance  models  were 
inadequate  in  explaining  the  behaviour  of  the  system,  and  the  same 
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conclusion  must  apply  to  these  subjectively  selected  models.  However,  if 
our  purpose  was  to  screen  the  variables  originally  proposed  prior  to  further 
experimentation,  a  selection  technique  such  as  this  which  is  able  to  quickly 
and  easily  determine  the  most  important  factors  would  be  highly  useful. 

3.14  Specific  Purpose  versus  All-Purpose  Models 

In  developing  models  thus  far,  we  have  specified  either  prediction  or 
explanation  as  the  purpose  for  the  models,  and  have  not  considered  whether 
the  resulting  models  could  adequately  serve  both  purposes.  The  first  point 
to  note  is  that  neither  of  the  specialised  sets  of  models  is  suitable  for  other 
than  its  intended  purpose.  The  prediction  models,  for  example,  contain  on 
average  over  three  times  as  many  terms  as  the  models  recommended  for 
explanation.  Such  a  large  number  of  terms,  some  of  which  are  not 
significant  even  at  the  30%  level,  would  make  an  explanatory  model 
excessively  complex,  and  greatly  hinder  meaningful  analysis.  The  5% 
significance  models  were  quickly  rejected  as  possible  prediction  models 
because  of  their  large  prediction  errors  relative  to  the  full  models.  The 
remaining  sets  of  models  examined  are  the  10%  significance  models  and  the 
near  minimum  MSE  models.  The  10%  models,  although  reasonably  well 
suited  for  explanation  are  not  suitable  for  prediction  because  of  the  21% 
mean  absolute  percentage  error  observed  in  Table  3.8  for  ail  factors  low  in 
the  attack  case.  The  near  minimum  MSE  models  were  found  to  be  almost 
as  capable  for  prediction  as  the  preferred  minimum  MSE  models,  but 


3.79 


average  over  twice  as  many  terms  as  the  explanatory  models,  and 
frequently  contain  terms  not  significant  at  the  20%  level,  and  are  therefore 
also  unsuitable  for  the  purpose  of  explanation.  Given  the  extent  of  the 
difference  between  the  specialised  models,  it  is  unlikely  that  a  good 
compromise  could  be  found  between  the  model  sizes  considered,  and  we 
therefore  conclude  that  for  this  data,  the  purposes  of  prediction  and 
explanation  are  incompatible,  and  specialised  models  are  required  for  each 
purpose. 


fiilfi  Summary 

This  chapter  has  completed  the  exploration  of  alternative  metamodels 
for  the  airbase  operability  simulation  and  a  brief  summary  of  the  main 
points  in  the  chapter  is  provided  below.  In  the  next  chapter,  the  lessons 
learned  from  the  exploration  are  presented,  followed  by  conclusions,  and 
recommendations  for  future  research. 


3.15.1  Existing  Database  and  Experimental  Design.  The  database 
and  the  experimental  design  from  the  previous  research  were  found  to 
restrict  the  development  of  new  models  to  the  same  polynomial  linear 
regression  form  used  in  the  previous  research. 


3.15.2  Unexpected  Regression  Results.  Preliminary  analysis  of  the 
original  models  revealed  unexpected  results  in  the  equivalence  of  different 
regression  techniques,  and  the  constancy  of  the  intercept  and  regression 
coefficients  between  models  containing  different  numbers  of  variables.  The 
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results  are  attributable  to  the  orthogonal  experimental  design,  and  although 
the  behaviour  is  documented,  it  receives  little  more  than  a  passing  mention 
in  the  literature  consulted. 

3.15.3  Design  Implications.  The  orthogonality  of  the  experimental 
design  was  found  to  greatly  narrow  the  possibilities  in  developing  new 
models.  The  techniques  generally  recommended  to  find  the  best 
combination  of  variables  to  include  in  a  model  were  found  to  be  largely 
inapplicable  because  the  variables  all  have  independent  effects.  Stepwise 
regression,  specifically  forward  selection,  was  determined  to  be  completely 
effective  in  finding  the  best  subset  of  variables  for  any  significance  level, 
while  adapted  best  subsets  techniques  were  shown  to  be  practical  for 
exploratory  model  development. 

3.15.4  Prediction  Models.  Two  sets  of  models  for  prediction  were 
developed  and  compared  with  both  the  full  models  and  the  baseline  10% 
significance  models.  The  best  models  for  prediction  were  found  to  contain 
substantially  more  terms  than  the  baseline  models,  and  were  shown  to  have 
less  bias  when  compared  with  the  full  models,  and  less  variance.  The  full 
models,  while  completely  unbiased,  were  shown  to  be  unsuitable  for 
prediction  because  of  the  excess  variance  caused  by  the  inclusion  of 
numerous  insignificant  terms. 

3.15.5  Explanatory  Models.  Two  techniques  were  proposed  for 
developing  explanatory  models:  one  based  on  the  significance  level  of  the 
variables  included  in  the  models;  and  another  based  on  a  subjective 
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assessment  of  the  relative  importance  of  the  candidate  variables.  Forward 
stepwise  selection  was  used  to  develop  models  with  terms  significant  at  the 
5%  and  the  1%  levels.  Comparison  with  the  baseline  models  suggested  that 
the  1%  significance  level  was  too  low,  and  that  too  much  information  was 
lost  The  5%  significance  models,  however,  were  considered  to  contain 
substantially  the  same  information  as  the  10%  models,  but  with  fewer 
spurious  terms  to  complicate  the  analysis.  The  5%  models  are  therefore 
recommended  for  the  purpose  of  explanation.  Models  developed  using  an 
assessment  of  the  relative  importance  of  the  variables  proved  to  be 
practically  identical  to  the  1%  significance  models,  and  were  also  rejected 
for  explanation  purposes. 


a 
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IV.  Conclusions.  Lessons  Learned,  and  Re 
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endations 


4.1  .Iateadaction 

This  chapter  presents  conclusions  regarding  the  exploratory  research 
into  alternative  metamodels  for  Diener's  airbase  operability  simulation 
(1989).  The  lessons  learned  during  the  exploration  are  also  presented,  and 
are  mainly  concerned  with  what  can  be  expected  in  developing  metamodels 
from  a  large  scale  simulation  experiment,  what  techniques  are  appropriate 
and  work  well,  and  what  techniques  are  of  less  use  and  should  be  avoided. 
Also  included  in  this  chapter  are  recommendations  for  further  research. 

4.2  CoaduaionB 

In  this  section,  the  research  objectives  stated  in  Chapter  I  are 
addressed  and  overall  conclusions  regarding  the  research  are  presented. 

4.2.1  Research  Objective.  The  research  objective  stated  in  Chapter  I 
was  to  investigate  whether  alternative  metamodels  other  than  those  derived 
by  Diener  can  be  used  to  effectively  represent  the  results  of  Diener's 
simulation.  To  achieve  the  objective,  three  questions  were  posed. 

4.2. 1.1  Question  One.  What  is  the  purpose  of  the  metamodel? 
For  example,  is  understanding  general  relationships  in  the  system  as 
simulated  the  primary  goal,  or  do  we  wish  to  make  predictions  about  the 
response  of  the  simulation  under  different  conditions?  Do  different  goals 
require  different  models? 
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What  are  the  important  criteria  in 


determining  the  suitability  of  a  metamodel?  For  example,  is  the  overall  fit 
of  the  model  the  primary  criterion,  or  are  there  other  important  factors  to 
consider? 

4.2. 1.3  Question  Three.  How  does  the  nature  of  the  output 
data,  and  the  experimental  design  on  which  it  is  based,  limit  or  restrict  the 
types  of  metamodels  that  can  be  developed? 

4.2.2  Conclusions  -  Question  One.  The  investigation  showed  that  for 
large  scale  simulations  such  as  Diener's  airbase  operability  simulation,  the 
purpose  of  a  metamodel  is  of  critical  importance  to  the  development  of  a 
metamodel  and  the  choice  of  variables  for  inclusion  in  the  model.  Separate 
models  were  developed  for  explanation  and  prediction,  and  neither  was 
found  to  be  suitable  for  the  other  purpose.  Single  all-purpose  models  were 
considered,  but  found  to  be  a  poor  compromise  for  the  specialised  models. 

4.2.3  Conclusions  -  Question  Two.  The  important  criteria  for 
determining  the  suitability  of  a  metamodel  were  found  to  depend  upon  the 
purpose  for  which  the  model  was  developed.  Explanatory  models 
concentrate  on  finding  the  least  number  of  variables  which  are  best  able  to 
explain  the  important  relationships  in  the  system.  The  criteria  for 
accepting  such  models  may  be  reasonably  objective,  for  example, 
predetermining  a  significance  level  for  the  model  and  accepting  the 
outcome,  or  may  be  more  subjective,  whereby  the  analyst  may  compare 
several  models  and  make  a  judgement  about  which  makes  the  best 
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compromise  between  simplicity  and  thorough  explanation.  Goodness  of  fit, 
measured  by  R3,  is  used  less  as  a  criterion  for  suitability  than  as  a  measure 
of  performance.  Judging  the  suitability  of  predictive  models,  on  the  other 
hand,  can  be  done  more  objectively.  Predictive  models  seek  a  balance 
between  low  bias  and  low  variance  of  a  prediction,  both  of  which  may  be 
calculated.  Although  a  compromise  must  be  made  because  bias  and 
variance  reach  their  minima  in  different  sized  models,  comparison  with 
acceptable  tolerances  can  make  the  compromise  more  objective.  Goodness  of 
fit  will  be  better  than  for  explanatory  models,  and  may  prove  to  be  a  useful 
criterion  for  suitability  if  the  adjusted  R2  measure  is  maximised. 

4.2.4  Conclusions  -  Question  Three.  The  existing  database  and  the 
experimental  design  on  which  it  is  based  was  found  to  be  very  restrictive, 
more  so  than  had  been  expected,  on  the  form  of  metamodel  possible  and  the 
techniques  available  to  derive  the  models.  The  metamodel  form  was  limited 
to  the  same  polynomial  form  proposed  by  Diener,  and  linear  least  squares 
regression  was  found  to  be  the  only  appropriate  technique  for  deriving  the 
models. 

4.2.5  CondusionB  -  Research  Objective  Overall.  Only  partial  success 
can  be  claimed  in  developing  alternative  metamodels  for  the  airbase 
operability  simulation.  The  models  developed  for  prediction,  while  of  the 
same  functional  form  as  the  existing  models,  are  significantly  different  in 
the  number  of  terms  included,  and  can  be  shown  to  achieve  better  results  in 
predicting  the  response  of  the  simulation,  both  in  terms  of  bias  and  the 
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variance  of  the  response.  The  models  suggested  for  explanation,  however, 
do  not  differ  greatly  from  the  existing  models,  and  without  deeper 
knowledge  of  the  airbase  operability  problem  it  is  inappropriate  to  suggest 
that  the  preferred  models  in  this  research  are  a  better  alternative  to  the 
existing  models.  An  important  positive  aspect  of  the  similarity  between  the 
new  and  existing  models  is  that  this  research  has  shown  that  there  are  no 
unusual  models  not  discovered  in  the  original  research  that  could  upset  the 
findings  of  that  research. 

4.2.6  Other  Achievements.  Although  the  goal  of  developing 
alternative  metamodels  was  only  partially  achieved,  the  exploration  process 
has  provided  a  number  of  lessons  which  it  is  hoped  will  be  of  use  to 
researchers  working  on  similar  problems.  The  lessons  learned  are 
presented  in  the  following  sections. 

4.3  Lessons  Related  to  Experimental  Design 

Several  important  lessons  learned  in  this  research  are  related  to  the 
experimental  design  chosen  in  the  previous  research.  Understanding  the 
design  and  its  properties  is  essential  if  metamodels  are  to  be  successfully 
developed. 

4.3.1  Model  Form.  The  experimental  design  was  found  to  limit  the 
model  form  to  polynomial  linear  regression  models.  This  limitation  will 
apply  generally  to  any  design  that  uses  qualitative,  or  indicator,  variables 
at  only  two  levels.  The  two  levels  of  the  variables  cannot  define  any  other 


than  linear  relationships  between  the  dependent  variable  and  the 
independent  variables,  so  that  quadratic  or  higher  terms  are  meaningless. 
The  inclusion  of  interaction  terms  is  another  aspect  of  model  form  that  is 
dependent  on  the  experimental  design.  When  a  fractional  factorial  design  is 
used,  the  resolution  of  the  design  dictates  whether  or  not  interaction  effects 
are  confounded  with  other  interaction  effects  or  even  main  effects.  The 
Resolution  V  design  used  in  the  prior  research  did  not  confound  any  main 
effects  or  their  first  order  (two-way)  interactions,  but  we  need  to  be  aware  of 
the  capabilities  of  an  experimental  design  to  ensure  that  an  inappropriate 
model  form  is  not  specified. 

4.3.2  Design  Matrix  Properties.  A  particularly  important  lesson  from 
this  research  is  that  the  properties  of  the  design  matrix  depend  upon  how  it 
is  coded.  However,  before  examining  the  properties  of  the  design  matrix, 
the  experimental  design  and  the  design  matrix  should  be  distinguished. 

The  experimental  design  determines  the  number  of  simulation  runs 
required,  and  the  combination  of  input  factors  for  each  run.  For  example, 
Diener's  experimental  design  requires  128  different  runs,  with  the  first 
simulation  run  to  be  carried  out  with  all  factors  at  their  low  level  (1989:45). 
The  design  matrix  allows  us  to  numerically  represent  the  level  of  each 
variable  for  each  run,  so  that  we  have  a  value  of  each  independent  variable 
for  regression  analysis.  The  coding  of  the  design  matrix  is  the  numerical 
value  we  assign  to  the  low  and  high  levels  of  the  variables,  but  the 
underlying  experimental  design  is  the  same,  regardless  of  how  the  variables 


are  coded.  The  properties  of  the  design  matrix  in  which  we  are  interested 
are  correlation  between  the  independent  variables,  and  whether  or  not  the 
matrix  is  orthogonal.  For  the  data  in  this  research,  when  the  low  level  is  , 

coded  as  0  and  the  high  level  as  1,  there  is  no  correlation  between  the 
independent  variables,  and  the  design  matrix  is  not  orthogonal.  When  the 
low  level  code  is  changed  to  -1,  however,  there  is  still  no  correlation  between 
the  variables,  but  the  design  matrix  is  now  orthogonal.  The  effects  of 
orthogonality  and  why  it  is  highly  desirable  will  be  discussed  shortly.  This 
lesson  is  important  because  coding  the  design  matrix  is  a  choice  usually  left 
to  the  researcher,  and  in  the  absence  of  dear  guidance  we  may  well  choose 
a  (0,1)  coding  scheme  because  model  analysis  can  be  simpler.  For  example, 
variables  effectively  drop  out  of  a  (0,1)  coded  model  when  set  to  their  low 
level.  The  key  point  is  that  before  choosing  a  coding  scheme,  the  properties 
of  the  design  matrix  with  that  scheme  should  be  confirmed.  As  the  only 
practical  choices  of  coding  scheme  for  a  two  level  design  are  (0,1)  and  (-1,1), 
checking  the  properties  of  the  design  matrix  for  both  schemes  is  feasible.  A 
design  matrix,  X,  is  orthogonal  if  XX  is  a  diagonal  matrix,  that  is,  all  terms 
except  on  the  main  diagonal  are  zero.  For  an  orthogonal  (-1,1)  coded  design 
matrix,  the  terms  on  the  main  diagonal  will  all  be  equal  to  the  number  of 
rows,  that  is,  design  points,  in  the  design  matrix.  • 

4.3.3  Design  Matrix  Effect  on  Regression  Analysis.  A  lesson  related 
to  the  properties  of  the  design  matrix  is  the  effect  that  those  properties  ' 

have  on  the  regression  analysis  used  to  develop  metamodels.  An  orthogonal 
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design  matrix  is  highly  desirable  because  it  ensures  that  every  variable 
makes  a  completely  independent  contribution  to  the  explanation  of  variance 
in  the  model,  thus  greatly  simplifying  the  development  and  analysis  of  the 
regression  models.  Analysis  is  simplified  because  with  no  correlation 
between  the  variables,  there  is  also  no  covariance,  and  the  calculation  of 
model  variance  reduces  to  the  sum  of  the  predictor  variances.  Additionally, 
because  the  terms  on  the  diagonal  of  the  X’X  matrix  are  all  equal,  the 
predictor  variances  are  also  all  equal.  Finally,  because  the  variables  each 
have  an  independent  effect  on  the  model,  the  coefficients  of  variables  that 
are  in  a  model  will  not  change  as  other  variables  are  added  to  or  removed 
from  the  model. 

4.3.4  An  Initial  Strategy.  From  the  discussion  above,  a  first  step 
prior  to  starting  regression  is  to  confirm  the  properties  of  the  chosen  design 
matrix.  A  useful  second  step  is  to  ensure  that  the  data  will  behave  as 
suggested  by  the  properties  of  the  design  matrix  by  examining  the  four 
types  of  sums  of  squares  described  in  Chapter  in.  Such  an  examination  can 
be  readily  accomplished  using  the  SAS  GLM  procedure,  and  requesting  as 
an  option  all  four  types  of  sums  of  squares. 

4.4  Regression  Model  Development 

Several  lessons  in  regression  model  development  arise  from  the 
independent  and  constant  effect  that  variables  have  when  the  design  matrix 
is  orthogonal.  We  find  that  automated  techniques  are  affected,  and  that  a 
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manual  technique  for  selecting  models  may  in  feet  be  preferable  to  a  range 
of  automated  techniques. 


;Tu  TomThk*  to  o:«  ™ 


The  first  lesson 


relating  to  model  development  is  that  a  simple  forward  selection  process 
will  always  find  the  best  model  for  a  given  significance  level.  The  forward 
selection  process  begins  with  no  variables  in  the  model  and  adds  the  most 
significant  variable  not  in  the  model,  continuing  until  none  of  the  remaining 
variables  are  significant  at  the  chosen  level.  It  is  stressed  that  this  lesson 
applies  to  a  designed  experiment  with  an  orthogonal  design  matrix  and 
uncorrelated  independent  variables.  The  second  lesson  is  that  for  a 
particular  selection  criterion,  the  same  model  will  result  regardless  of  the 
regression  technique  used.  Both  these  lessons  can  make  a  significant 
difference  to  the  way  the  regression  analysis  of  a  large  scale  experiment  is 
carried  out.  First,  conventional  wisdom  usually  suggests  that  using 
automatic  model  selection  techniques  carries  the  risk  that  some  unusual 


combination  of  variables  might  not  be  detected,  and  that  the  unusual 
combination  could  greatly  influence  our  results.  When  we  know  that  such  a 
risk  is  absent,  we  can  rely  on  the  efficiency  and  convenience  of  an  automatic 
search  and  be  confident  that  the  resulting  models  are  valid.  Second,  if  all 
automatic  techniques  are  equally  effective,  then  the  most  efficient  should  be 
used.  In  selecting  from  a  large  number  of  possible  variables  if  will  usually 
be  the  case  that  relatively  few  will  be  included,  so  a  forward  selection 
technique  is  preferable  to  backward  elimination  because  forward  selection 
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requires  fewer  steps,  and  thus  takes  less  time  and  produces  less  output  to 
analyse.  As  we  saw  in  this  research,  the  difference  can  be  very  substantial 
with  one  example  where  forward  selection  was  complete  after  one  step, 
while  backward  elimination  took  54  steps  to  reach  the  same  conclusion.  A 
stepwise  procedure  which  both  selects  and  eliminates  variables  reduces  to  a 
forward  selection,  because  terms  will  never  be  eliminated  once  they  have 
been  added,  so  again,  the  simple  forward  selection  technique  is  preferred. 

4.4.2  Other  Computerised  Model  Selection  Techniques.  Although 
forward  selection  will  be  effective  in  finding  a  model  for  a  specific 
significance  level  we  have  already  seen  that  the  commonly  used  significance 
levels  can  be  arbitrary.  Also,  evaluation  of  a  range  of  model  sizes  is 
cumbersome.  Two  techniques,  only  one  of  which  proved  to  be  useful,  were 
used  in  an  attempt  to  avoid  the  limitations  of  forward  selection  when 
searching  for  models  based  on  criteria  other  than  significance  level.  Both 
techniques  were  applied  based  on  the  premise  that  for  this  type  of  data, 
there  is  no  benefit  in  considering  other  than  the  best  model  of  a  particular 
size. 

4.4.2. 1  The  SAS  RSQUARE  Procedure.  The  technique  found 
to  be  useful  is  the  SAS  RSQUARE  procedure,  which  is  able  to  provide  a 
selection  of  statistics  on  the  desired  range  of  model  sizes  in  an  easily 
comparable  form.  RSQUARE  was  most  useful  in  finding  the  models  with 
maximum  adjusted  R3.  Important  considerations  for  the  use  of  the 
RSQUARE  procedure  are  that  only  the  single  best  subset  model  of  a  given 
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size  should  be  requested;  and  that  the  maximum  number  of  variables  that 
the  procedure  can  accept  appears  to  be  approximately  fifty.  The  size 
limitation  was  avoided  in  this  research  by  not  including  the  least  significant 
variables  in  the  pool  of  variables  for  selection,  as  the  model  sizes  considered 
would  never  include  these  variables.  A  shortcoming,  however,  of  reducing 
the  pool  of  variables  so  that  the  RSQUARE  procedure  can  be  used  is  that 
some  statistics,  notably  Mallow's  Cp,  are  reported  incorrectly  because  the 
procedure  incorrectly  assumes  that  the  full  model  is  the  truncated  pool  of 
variables.  Although,  for  reasons  that  will  be  discussed  shortly,  Mallow's  Cp 
was  found  to  be  of  no  use  in  this  research,  it  is  possible  that  in  other  cases 
Cp  could  be  useful  and  therefore  reducing  the  pool  of  variables  would  be 
unacceptable. 

4.4.2.2  The  SAS  Maximum  Ra  Procedure-  The  SAS  Maximum 
R2  procedure,  although  capable  of  quickly  finding  the  best  model  for  all 
model  sizes,  was  not  found  to  be  particularly  useful  in  this  research  because 
its  unwieldy  standard  output  made  comparisons  between  different  models 
difficult.  Maximum  R2  is  designed  to  overcome  the  limitations  of  stepwise 
regression  when  unusual  variable  combinations  affect  the  regression  results, 
but  in  a  dataset  such  as  this,  the  procedure’s  capabilities  are  not  required. 
The  manual  search  technique  described  in  the  previous  chapter  can  provide 
the  same  information  as  maximum  R2,  but  in  a  more  convenient  form. 
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A  common  problem  with  regression  analysis  involving  many  variables 
»  is  that  the  number  of  possible  models  grows  exponentially  as  the  number  of 
variables  increases,  so  that  for  the  data  in  this  research,  2“,  or 
*  approximately  3.6  x  101S  different  models  could  be  formed  from  the  ten 

independent  variables  and  their  45  interactions.  Obviously  an  exhaustive 
search  of  so  many  models  is  impossible.  A  very  useful  lesson  from  this 
research  is  that  when  the  variables  make  an  independent  contribution  to  a 
model,  the  number  of  models  that  are  of  practical  interest  reduces  to  just 
the  number  of  variables,  including  interactions.  An  exhaustive  search  of, 
for  this  research,  55  models  is  a  much  more  practical  proposition,  especially 
if  the  information  about  those  models  can  be  presented  in  a  readily 
comparable  fashion.  The  manual  search  technique  introduced  in  the 
previous  chapter  achieves  this  goal  because  it  provides  in  a  single  worksheet 
nearly  all  the  information  required  to  choose  the  variables  to  include  in  a 
model.  The  only  additional  information  required  is  the  probability 
associated  with  the  value  of  the  t  statistic.  As  software  or  tables  are  readily 
available  to  calculate  the  probabilities,  which  represent  the  significance 
level  of  the  last  variable  to  enter  the  model,  all  the  information  to  select  a 
model  is  available.  A  sample  worksheet  and  the  corresponding  p-values  are 
provided  in  Appendix  E,  in  Tables  E.l  and  E.2  respectively. 

4.5.1  Model  Selection.  Models  can  be  selected  on  the  basis  of 
significance  by  choosing  the  largest  model  with  a  probability  for  the  t 
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statistic  less  than  or  equal  to  title  desired  significance  level.  Other  selection 
methods  that  are  easily  used  include  selection  of  a  fixed  number  of 
variables,  subjective  selections  as  used  in  the  previous  chapter,  or  selection 
of  variables  with  regard  to  maximum  adjusted  Ra,  which  is  equivalent  to 
minimum  MSE.  Although  this  particular  lesson  was  learned  too  late  to 
greatly  assist  the  present  research,  we  can  see  that  when  the  data  behaves 
in  a  fashion  similar  to  the  data  in  this  research,  the  model  selection  process 
can  be  reduced  to  choosing  the  model  that  satisfies  the  desired  criteria  from 
a  spreadsheet  table. 

4.5.2  Advantages.  Some  advantages  of  using  this  technique  are  that 
all  models  of  practical  interest  can  be  assessed  simultaneously,  that  several 
selection  criteria  can  be  applied  together,  and  that  the  chosen  model's 
performance  on  other  than  the  selection  criterion  can  be  evaluated.  Possibly 
the  greatest  advantages,  however,  are  that  the  analyst's  judgement  can 
replace  the  inflexibility  of  a  computerised  technique,  and  that  numerous 
computer  runs  and  the  subsequent  analysis  of  their  output  can  be  avoided 
altogether.  The  only  computer  runs  required  are  the  initial  regression  of 
the  full  model  to  provide  the  data  for  the  table,  and  a  final  run  with  the 
selected  model  to  provide  residual  data,  and  to  present  the  model  and  its 
analysis  of  variance  table  in  more  conventional  form.  Other  aspects  of  this 
manual  technique  that  could  prove  useful  are  the  ease  of  producing  visual 
aids  to  regression  analysis  from  the  sorted  data.  The  graphs  in  Appendices 
H  and  I  showing  regression  coefficients  and  partial  R3  for  models  with  up  to 


twenty  variables  are  one  example.  Plots  of  adjusted  R2  or  Cp  against  the 
number  of  predictors,  both  of  which  are  often  recommended  for  model 
selection,  are  other  visual  aids  that  could  be  easily  produced  from  the 
spreadsheet. 

4  fi.a  Bjaadsaatagfifl*  The  main  disadvantage  of  using  the  manual 
search  technique  is  the  overhead  in  importing  and  sorting  the  full  model 
information  and  setting  up  worksheets.  For  simple  analysis  where  the 
criterion  for  model  selection  is  well  defined,  and  only  a  single  model  needs 
to  be  produced,  the  overhead  is  probably  not  justified.  A  minor 
disadvantage  is  the  requirement  for  a  spreadsheet  with  reasonably  powerful 
data  importing  and  sorting  capabilities. 

4.6  Model  Selection  Criteria 

A  number  of  problems  were  encountered  with  the  application  of 
several  model  selection  criteria,  specifically  adjusted  R2  and  Mallow's  Cp 
statistic.  Without  another  dataset  for  comparison,  it  is  considered  likely  but 
not  certain  that  these  problems  stem  from  the  very  large  number  of 
observations  relative  to  the  number  of  variables  included  in  the  models. 

4.6.1  Adjusted  R2.  Adjusted  R2  proved  to  be  useful  only  for  selecting 
models  for  prediction  purposes,  as  the  statistic  did  not  reach  a  maximum 
until  variables  were  significant  at  typically  the  30%  level.  From  Equation 
3.9  we  see  that  when  n  is  large  relative  to  p,  only  small  reductions  in  SSE 
are  needed  for  adjusted  R2  to  continue  increasing.  We  can  therefore  expect 
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that  in  an  experiment  with  many  observations,  the  use  of  adjusted  Ra  as  a 
model  selection  criterion  will  result  in  models  with  relatively  many  terms, 
some  of  which  will  be  largely  magnificent. 

4.fi.2  Mnllnw'a  f!  Mallow's  Cp  statistic  was  found  to  be  of  no  use  in 
this  research  because,  for  the  model  sizes  of  interest,  the  statistic  often  took 
a  negative  value,  a  possibility  not  mentioned  in  the  literature  on  the  topic. 

The  reason  for  this  behaviour  is  not  as  dear  as  for  adjusted  Ra,  nor  can  we 
generalise  about  the  behaviour  of  Cp  in  other  experiments.  From  Equation 
3.10  the  large  number  of  observations  relative  to  model  size  has  a  major 
influence  in  reducing  Cp  below  zero,  but  the  ratio  of  SSE  to  MSE  depends 
strongly  on  the  individual  data.  The  best  we  can  do  is  point  out  that  it  is 
possible  for  Cp  to  be  negative  when  there  are  many  observations  but 
relatively  few  terms  in  a  model. 

4.7  Purpose  of  the  Metamodda 

On  a  number  of  occasions  during  this  research,  we  have  highlighted 
the  difference  between  the  two  main  purposes  of  simulation  metamodels; 
that  is,  use  of  the  metamodels  to  predict  the  response  of  the  simulation  to  a 
particular  set  of  input  conditions;  and  use  of  the  metamodels  to  understand 
and  explain  the  primary  relationships  between  the  simulation  response  and  , 

the  independent  variables.  Indeed,  we  have  shown  that  for  the  data  in  this 
research,  the  two  purposes  are  not  compatible,  and  require  distinctly  * 

different  models.  While  it  may  not  necessarily  be  the  case  in  all  situations 


4.14 


that  different  models  are  required  for  the  two  purposes,  it  is  more  likely 
that  the  forms  of  the  models  will  diverge  as  the  number  of  variables  and 
interactions  that  are  considered  increase.  Several  lessons  that  emerge  from 
the  distinction  between  the  purposes  are  discussed  below. 

4.7.1  Prediction.  When  developing  models  for  prediction,  the 
primary  consideration  is  finding  a  model  which  results  in  an  accurate 
prediction  with  minimal  variance.  Thus,  the  specific  terms  that  are 
included  in  a  predictive  model  are  of  much  less  concern  than  their  aggregate 
effect  on  the  prediction.  The  first  lesson  to  arise  is  that  because  variance  as 
well  as  bias  must  be  considered  it  is  likely  that  the  full  model,  which  is 
unbiased,  will  not  be  useful  for  prediction  because  of  the  large  variance 
involved.  This  observation  should  hold  for  most  experiments  where  a 
relatively  large  number  of  variables  make  very  small  predictive 
contributions  to  their  models  while  still  adding  variance.  The  implication, 
therefore,  is  that  a  tradeoff  is  required  between  the  amount  of  bias  in  a 
reduced  model  and  its  prediction  variance.  The  acceptable  level  of  bias  and 
the  desired  precision  of  a  prediction  will  be  different  in  every  experiment, 
and  cannot  be  generalised.  The  second  lesson  is  that  calculation  of  the 
variance  is  surprisingly  straightforward  when  the  design  is  orthogonal.  We 
saw  in  Chapter  HI  that  no  covariance  is  present,  so  summing  the  individual 
variances  provides  an  estimate  for  the  variance  of  the  expected  value  of  the 
response.  Adding  MSE  to  the  sum  of  the  variances  provides  an  estimate  for 
the  variance  of  a  new  prediction.  Because  the  two  aspects  of  prediction. 
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that  is,  predicting  a  new  response,  or  estimating  the  expected  value  of  the 
response,  have  different  variance  calculations,  it  is  possible  that  different 
models  could  be  appropriate  depending  upon  the  specific  purpose.  In  either  # 

case,  an  understanding  of  the  behaviour  of  the  individual  predictor 
variances,  their  sum,  and  MSE  as  the  number  of  terms  in  a  model  varies  is 
essential  to  developing  good  predictive  models. 

4.7.2  Explanation  The  key  lesson  in  developing  models  for 
explanation  is  the  degree  of  subjective  assessment  required  in  striking  a 
balance  between  parsimony  and  ensuring  that  all  important  effects  have 
been  captured.  Unlike  predictive  models,  we  are  highly  interested  in 
individual  variables,  the  exclusion  or  inclusion  of  which  could  make  a  large 
difference  to  our  interpretation  of  the  model.  To  reduce  the  impact  of 
subjectivity,  it  is  recommended  that  a  range  of  models  be  developed  and 
compared  before  choosing  a  final  explanatory  model. 

4.8  Lessons  Learned  Summary 

The  lessons  described  above  highlight  various  aspects  of  developing 
regression  metamodels  that  are  likely  to  confront  a  researcher.  None  of  the 
lessons  are  particularly  profound,  but  taken  as  a  whole,  they  have  the 
potential  to  ease  the  model  development  and  analysis  task  for  similar  data. 

An  underlying  theme  of  the  lessons  learned  in  this  research  has  been  the 

effect  of  an  orthogonal  design  matrix  on  the  analysis  of  large  multiple  • 

regression  models,  and  it  must  be  stressed  that  these  lessons  are  applicable 
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only  to  data  meeting  the  dual  requirements  of  an  orthogonal  design  matrix, 
and  no  correlation  among  the  independent  variables. 


1 Q  Rficnmmendfttions  for  Further  Research 

Several  areas  for  possible  further  research  are  evident,  including 
examination  of  similar  data  to  the  data  used  in  this  research,  some 
experimental  design  issues,  and  the  time  dependent  aspect  of  the 
metamodels. 

id LI  Examination  nf  Similar  Data.  The  techniques  proposed  during 
this  research  and  highlighted  in  the  lessons  learned  depend  to  a  large 
extent  on  the  independent  behaviour  of  the  variables  in  a  regression  model. 
Further  research  is  required  to  establish  that  the  behaviour  of  the  variables 
in  this  research  is  common  to  other  large  scale  designed  experiments  using 
similar  orthogonal  designs.  A  related  issue  is  to  establish  whether  the 
manual  search  technique  outlined  in  this  research  has  practical  application 
to  other  problems. 

4.9.2  Experimental  Design  Issues.  This  research  found  that  the 
existing  experimental  design  was  limiting  with  regard  to  model  form  and 
model  development  techniques.  Future  research  could  examine  the 
alternate  experimental  designs,  with  the  aim  of  possibly  finding  an 
alternate  functional  form  for  the  models.  Clearly,  such  future  research  is  a 
major  undertaking,  because  the  simulations  would  have  to  be  repeated 
under  the  new  designs.  An  extension  to  the  current  design  would  be  to 
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introduce  a  third  level  for  the  variables  identified  as  important  in  the 

existing  models,  and  then  use  response  surface  methodology  to  derive 

metamodels,  while  a  completely  different  form  of  design  such  as  the  , 

frequency  domain  experiments  described  by  Starbird  (1990)  may  be  a 

possibility.  It  is  likely  that  any  alternative  experimental  design  would  be 

more  complex  than  the  two  level  design  used  by  Diener,  so  the  number  of 

variables  considered  in  Diener' s  original  research  could  become 

unmanageable.  A  possible  solution  could  be  to  consider  the  existing 

research  as  a  screening  experiment  in  which  the  existing  metamodels  have 

identified  the  important  variables  and  interactions  which  thus  merit  further 

attention  in  more  complex  designs. 

4.9.3  Time  Dependent  Aspects.  It  is  dear  from  the  metamodels  that 
the  effects  that  some  variables  have  on  sortie  generation  vary  from  day  to 
day,  but  we  are  unable  to  represent  this  time  dependent  behaviour  in  the 
existing  models.  We  saw  in  Chapter  II  that  other  research  with  time  series 
output  used  a  single  value  to  characterise  the  time  series  (Kleijnen  et  al., 

1979).  Such  an  approach  was  briefly  considered  for  the  data  in  this 

research,  but  seems  inadequate  for  a  relatively  short  time  series  in  which 

each  element  is  important.  Using  a  single  value  for  each  time  series  has 

the  effect  of  collapsing  all  thirty  metamodels  into  one  model,  but  still  does  , 

not  tell  us  anything  about  the  effect  of  a  variable  at  a  given  time. 

Incorporating  the  time  dimension  into  some  sort  of  model  is  dearly  a  highly  • 
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Attack  Case  Max  Adj  Rsqr  Day  1 
*  Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

11  29849. 

96875  2713.63352 

27 

.210 

0.0001 

Error 

116  11568. 

50000  99.72845 

C  Total 

127  41418. 

46875 

Root 

MSE 

9.98641 

R-square 

0.7207 

Dep  Mean 

89.60938 

Adj  R-aq 

0.6942 

C.V. 

11.14438 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter^ 

Prob 

»  >  ITI 

INTERCEP 

1 

89.609375 

0.88268256 

101.519 

0.0001 

B1 

1 

-10.171875 

2.33535854 

-4.356 

0.0001 

B2 

1 

3.453125 

2.33535854 

1.479 

0.1420 

B3 

1 

-13.359375 

2.33535854 

-5.720 

0.0001 

B4 

1 

2.828125 

2.33535854 

1.211 

0.2284 

B5 

1 

16.828125 

2.33535854 

7.206 

0.0001 

B6 

1 

8.703125 

2.33535854 

3.727 

0.0003 

B7 

1 

-2.609375 

2.33535854 

-1.117 

0.2662 

D 

1 

11.953125 

0.88268256 

13.542 

0.0001 

BH 

1 

1.046875 

0.88268256 

1.186 

0.2380 

JK 

1 

-0.953125 

0.88268256 

-1.080 

0.2825 

GK 

1 

-0.921875 

0.88268256 

-1.044 

0.2985 

Attack  Casa  Max  Ad j  Raqr  Day  2 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  381325. 

25000  15253.01000 

60. 

428 

0.0001 

Error 

102  25746. 

62500  252.41789 

C 

Total 

127  407071. 

87500 

Root 

MSE 

15.88766 

R-square 

0.9368 

Dep  Mean 

83.21875 

Adj  R-aq 

0.9212 

C.V. 

19.09145 

Para 

meter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  | T I 

INTERCEP 

1 

83.218750 

1.40428444 

59.261 

0.0001 

B1 

1 

-6.531250 

3.71538739 

-1.758 

0.0818 

B2 

1 

57.281250 

3.71538739 

15.417 

0.0001 

B3 

1 

-29.406250 

3.71538739 

-7.915 

0.0001 

B4 

1 

6.031250 

3.71538739 

1.623 

0.1076 

B5 

1 

46.406250 

3.71538739 

12.490 

0.0001 

B6 

1 

-16.593750 

3.71538739 

-4.466 

0.0001 

B7 

1 

-35.406250 

3.71538739 

-9.530 

0.0001 

D 

1 

40.578125 

1.40428444 

28.896 

0.0001 

CJ 

1 

-11.812500 

1.40428444 

-8.412 

0.0001 

BH 

1 

8.875000 

1.40428444 

6.320 

0.0001 

CE 

1 

2.828125 

1.40428444 

2.014 

0.0467 

HJ 

1 

-2.812500 

1.40428444 

-2 . 003 

0.0479 

AG 

1 

2.765625 

1.40428444 

1.969 

0.0516 

BE 

1 

2.484375 

1.40428444 

1.769 

0.0799 

E 

1 

2.203125 

1.40428444 

1.569 

0.1198 

EK 

1 

-2.125000 

1.40428444 

-1.513 

0.1333 

EJ 

1 

2.015625 

1.40428444 

1.435 

0.1542 

F 

1 

1.937500 

1.40428444 

1.380 

0.1707 

OE 

1 

1.906250 

1.40428444 

1.357 

0.1776 

EH 

1 

1.796875 

1.40428444 

1.280 

0.2036 

DG 

1 

-1.781250 

1.40428444 

-1.268 

0.2075 

HK 

1 

-1.640625 

1.40428444 

-1.168 

0.2454 

C 

1 

1.562500 

1.40428444 

1.113 

0.2685 

AB 

1 

-1.500000 

1.40428444 

-1.068 

0.2880 

AJ 

1 

1.468750 

1.40428444 

1.046 

0.2981 

♦ 


r 


Attack  Casa  Max  Adj  Rsqr  Day  3 


Analysis  of  Variance 


Stun  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

19  263118. 

75000  13848. 35526 

22 

.749 

0.0001 

Error 

108  65744. 

75000  608.74769 

C  Total 

127  328863. 

50000 

Root 

MSE 

24.67281 

R-square 

0.8001 

Dep  Mean 

104.31250 

Adj  R-sq 

0.7649 

C.V. 

23.65279 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter^ 

Prob 

|  >  |T| 

ZNTERCEP 

1 

104.312500 

2.18078914 

47.832 

0.0001 

B1 

1 

-9.750000 

5.76982574 

-1.690 

0.0939 

B2 

1 

20.312500 

5.76982574 

3.520 

0.0006 

B3 

1 

-23.625000 

5.76982574 

-4.095 

0.0001 

B4 

1 

28.312500 

5.76982574 

4.907 

0.0001 

BS 

1 

7.125000 

5.76982574 

1.235 

0.2196 

B6 

1 

63.875000 

5.76982574 

11.071 

0.0001 

B7 

1 

-34.375000 

5.76982574 

-5.958 

0.0001 

D 

1 

22.328125 

2.18078914 

10.239 

0.0001 

BH 

1 

14.968750 

2.18078914 

6.864 

0.0001 

CJ 

1 

-5.390625 

2.18078914 

-2.472 

0.0150 

E 

1 

3.812500 

2.18078914 

1.748 

0.0833 

AC 

1 

-2.953125 

2.18078914 

>1.354 

0.1785 

FJ 

1 

2.890625 

2.18078914 

1.325 

0.1878 

BE 

1 

2.750000 

2.18078914 

1.261 

0.2100 

BJ 

1 

-2.625000 

2.18078914 

-1.204 

0.2313 

AE 

1 

-2.625000 

2.18078914 

-1.204 

0.2313 

B 

1 

-2.593750 

2.18078914 

-1.189 

0.2369 

AK 

1 

-2.312500 

2.18078914 

-1.060 

0.2913 

A 

1 

2.250000 

2.18078914 

1.032 

0.3045 
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Attack  Casa  Max  Adj  Raqr  Day  4 


Analysis  of  Variance 


Source 

Sum  of 
DF  Squares 

Mean 

Square 

F  Value 

Prob>F 

Model 

Error 

C  Total 

30  81833.56250 

97  36286.65625 

127  118120.21875 

2727.78542 

374.08924 

7.292 

0.0001 

Root  MSE 
Dep  Mean 

C.V. 

19.34139  R-square 
92.32813  Adj  R-sq 
20.94853 

0.6928 

0.5978 

Parameter  Estimates 


Variable 

DF 

Parameter 

Estimate 

Standard 

Error 

T  for  HO: 
Parameters 

Prob  >  I T I 

XNTERCEP 

1 

92.328125 

1.70955321 

54.007 

0.0001 

B1 

1 

3.734375 

4.52305265 

0.826 

0.4110 

B2 

1 

0.296875 

4.52305265 

0.066 

0.9478 

B3 

1 

29.671875 

4.52305265 

6.560 

0.0001 

B4 

1 

9.359375 

4.52305265 

2.069 

0.0412 

B5 

1 

17.109375 

4.52305265 

3.783 

0.0003 

B6 

1 

-12.828125 

4.52305265 

-2.836 

0.0056 

B7 

1 

-12.890625 

4.52305265 

-2.850 

0.0053 

D 

1 

9.484375 

1.70955321 

5.548 

0.0001 

BH 

1 

6.390625 

1.70955321 

3.738 

0.0003 

EG 

1 

-4.703125 

1.70955321 

-2.751 

0.0071 

HK 

1 

-3.500000 

1.70955321 

-2.047 

0.0433 

G 

1* 

3.453125 

1 . 70955321 

2.020 

0.0462 

JK 

1 

-3.421875 

1.70955321 

-2.002 

0.0481 

AF 

1 

-3.062500 

1.70955321 

-1.791 

0.0763 

C 

1 

-3.062500 

1.70955321 

-1.791 

0.0763 

CF 

1 

-2.984375 

1.70955321 

-1.746 

0.0840 

E 

1 

2.828125 

1.70955321 

1.654 

0.1013 

FG 

1 

2.812500 

1.70955321 

1.645 

0.1032 

B 

1 

2.687500 

1.70955321 

1.572 

0.1192 

BD 

1 

2.593750 

1.70955321 

1.517 

0.1325 

CJ 

1 

-2.531250 

1.70955321 

-1.481 

0.1419 

DG 

1 

-2.328125 

1.70955321 

-1.362 

'  0.1764 

BJ 

1 

2.218750 

1.70955321 

1.298 

0.1974 

FH 

1 

2.171875 

1.70955321 

1.270 

0.2070 

DF 

1 

-2.062500 

1.70955321 

-1.206 

0.2306 

HJ 

1 

-2.031250 

1.70955321 

-1.188 

0.2377 

CG 

1 

-1.968750 

1.70955321 

-1.152 

0.2523 

CH 

1 

-1.953125 

1.70955321 

-1.142 

0.2561 

AG 

1 

-1.796875 

1.70955321 

-1.051 

0.2958 

F 

1 

1.781250 

1.70955321 

1.042 

0.3000 

Attack  Case  Max  Adj  Raqr  Day  5 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

28  117481. 

53125  4195.76897 

7 

.617 

0.0001 

Error 

99  54533. 

08594  550.83925 

C 

Total 

127  172014. 

61719 

Root 

MSE 

23.46996 

R-square 

0.6830 

Dep  Mean 

102.05469 

Adj  R-aq 

0.5933 

C.V. 

22.99744 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate* 

Error  Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

102.054688 

2.07447142 

49.196 

0.0001 

B1 

1 

7.132812 

5.48853547 

1.300 

0.1968 

B2 

1 

-22.367188 

5.48853547 

-4.075 

0.0001 

B3 

1 

53.257813 

5.48853547 

9.703 

0.0001 

B4 

1 

4.820312 

5.48853547 

0.878 

0.3819 

B5 

1 

-1.804688 

5.48853547 

-0.329 

0.7430 

B6 

1 

2.695312 

5.48853547 

0.491 

0.6245 

B7 

1 

-11.804688 

5.48853547 

-2.151 

0.0339 

B 

1 

8.726563 

2.07447142 

4.207 

0.0001 

D 

1 

6.945313 

2.07447142 

3.348 

0.0012 

BH 

1 

5.414063 

2.07447142 

2.610 

0.0105 

G 

1 

4.789063 

2.07447142 

2.309 

0.0230 

BF 

1 

-4.757813 

2.07447142 

-2.294 

0.0239 

E 

1 

4.148438 

2.07447142 

2.000 

0.0483 

HK 

1 

-3.945313 

2.07447142 

-1.902 

0.0601 

HJ 

1 

-3.835938 

2.07447142 

-1.849 

0.0674 

CE 

1 

-3.632813 

2.07447142 

-1.751 

0.0830 

AH 

1 

-3.335938 

2.07447142 

-1.608 

0.1110 

AG 

1 

-3.257813 

2.07447142 

-1.570 

0.1195 

K 

1 

2.867188 

2.07447142 

1.382 

0.1700 

J 

1 

2.695313 

2.07447142 

1.299 

0.1969 

CH 

1 

-2.695313 

2.07447142 

-1.299 

0.1969 

BE 

1 

-2.585938 

2.07447142 

-1.247 

0.2155 

AK 

1 

-2.492188 

2.07447142 

-1.201 

0.2325 

FJ 

1 

-2.414063 

2.07447142 

-1.164 

0.2473 

BC 

1 

2.382813 

2.07447142 

1.149 

0.2535 

DJ 

1 

-2.382813 

2.07447142 

-1.149 

0.2535 

CG 

1 

-2.335938 

2.07447142 

-1.126 

0.2629 

FG 

1 

2.304688 

2.07447142 

1.111 

0.2693 

% 
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Attack  Casa  Max  Adj  Raqr  Day  6 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DP  Squares  Square 

F  Value 

Prob>F 

Model 

26  115118. 

26563  4427.62560 

4. 

544 

0.0001 

Error 

101  98422. 

16406  974.47687 

C  Total 

127  213540. 

42969 

Root 

MSE 

31.21661 

R-square 

0.5391 

Dep  Mean 

166.22656 

Adj  R-sq 

0.4204 

C.V. 

18.77956 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DP 

Estimate 

Error  Parameter-0 

Prob  >  I T | 

INTERCEP 

1 

166.226563 

2.75918476 

60.245 

0.0001 

B1 

1 

-30.851563 

7.30011671 

-4.226 

0.0001 

B2 

1 

9.710938 

7.30011671 

1.330 

0.1864 

B3 

1 

15.960938 

7.30011671 

2.186 

0.0311 

B4 

1 

11.335938 

7.30011671 

1.553 

0.1236 

BS 

1 

14.835938 

7.30011671 

2.032 

0.0447 

B6 

1 

-8.476563 

7.30011671 

-1.161 

0.2483 

B7 

1 

-26.601563 

7.30011671 

-3.644 

0.0004 

6 

1 

11.304688 

2.75918476 

4.097 

0.0001 

B 

1 

10.789063 

2.75918476 

3.910 

0.0002 

HJ 

1 

-6.539063 

2.75918476 

-2.370 

0.0197 

BH 

1 

5.835938 

2.75918476 

2.115 

0.0369 

E 

1 

5.445313 

2.75918476 

1.974 

0.0512 

HK 

1 

-5.273438 

2.75918476 

-1.911 

0.0588 

JK 

1 

-5.226563 

2.75918476 

-1.894 

0.0611 

PJ 

1 

-5.101563 

2.75918476 

-1.849 

0.0674 

PG 

1 

4.757813 

2.75918476 

1.724 

0.0877 

BK 

1 

4.742188 

2.75918476 

1.719 

0.0887 

CG 

1 

-3.648438 

2.75918476 

-1.322 

0.1891 

A 

1 

3.632813 

2.75918476 

1.317 

0.1909 

DG 

1 

-3.554688 

2.75918476 

-1.288 

0.2006 

CE 

1 

-3.195313 

2.75918476 

-1.158 

0.2496 

FK 

1 

-3.148438 

2.75918476 

-1.141 

0.2565 

BJ 

1 

3.070313 

2.75918476 

1.113 

0.2685 

DE 

1 

-2.945313 

2.75918476 

-1.067 

0.2883 

AJ 

1 

2.914063 

2.75918476 

1.056 

0.2934 

AK 

1 

-2.851563 

2.75918476 

-1.033 

0.3038 
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Attack  Case  Max  Adj  Raqr  Day  7 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

32  112545. 

68750  3517.05273 

5. 

627 

0.0001 

Error 

95  59375. 

92969  625.00979 

C  Total 

127  171921. 

61719 

Root 

MSE 

25.00020 

R- square 

0.6546 

Dap  Mean 

148.55469 

Adj  R-sq 

0.5383 

C.V. 

16.82895 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob 

•  >  IT! 

INTERCEP 

1 

148.554688 

2.20972599 

67.228 

0.0001 

B1 

1 

-26.742188 

5.84638544 

-4.574 

0.0001 

B2 

1 

5.320313 

5.84638544 

0.910 

0.3651 

B3 

1 

23.007813 

5.84638544 

3.935 

0.0002 

B4 

1 

8.507813 

5.84638544 

1.455 

0.1489 

B5 

1 

17.382813 

5.84638544 

2.973 

0.0037 

B6 

1 

-6.117187 

5.84638544 

-1.046 

0.2981 

B7 

1 

-37.929688 

5.84638544 

-6.488 

0.0001 

B 

1 

10.554688 

2.20972599 

4.776 

0.0001 

G 

1 

8.960938 

2.20972599 

4.055 

0.0001 

HK 

1 

-5.351563 

2.20972599 

-2.422 

0.0173 

HJ 

1 

-4.570313 

2.20972599 

-2.068 

0.0413 

AH 

1 

-4.304688 

2.20972599 

-1.948 

0.0544 

H 

1 

3.945313 

2.20972599 

1.785 

0.0774 

BC 

1 

-3.914063 

2.20972599 

-1.771 

0.0797 

CJ 

1 

3.851563 

2.20972599 

1.743 

0.0846 

A 

1 

3.835938 

2.20972599 

1.736 

0.0858 

AK 

1 

-3.804688 

2.20972599 

-1.722 

0.0884 

EF 

1 

-3.226563 

2.20972599 

-1.460 

0.1475 

GJ 

1 

3.164063 

2.20972599 

1.432 

0.1555 

AD 

1 

3.148438 

2.20972599 

1.425 

0.1575 

CE 

1 

-3.101563 

2.20972599 

-1.404 

0.1637 

JK 

1 

-3.007813 

2.20972599 

-1.361 

0.1767 

BG 

1 

2.992188 

2.20972599 

1.354 

0.1789 

FG 

1 

2.992188 

2.20972599 

1.354 

0.1789 

AB 

1 

2.773438 

2.20972599 

1.255 

0.2125 

DK 

1 

-2.773438 

2.20972599 

-1.255 

0.2125 

GH 

1 

-2.742188 

2.20972599 

-1.241 

0.2177 

CG 

1 

-2.726563 

2.20972599 

-1.234 

0.2203 

AJ 

1 

-2.648438 

2.20972599 

-1.199 

0.2337 

FH 

1 

2.351563 

2.20972599 

1.064 

0.2899 

CH 

1 

2.257813 

2.20972599 

1.022 

0.3095 

AE 

1 

-2.226563 

2.20972599 

-1.008 

0.3162 
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Attack  Case  Max  Adj  Rsqr  Day  8 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

27  100693. 

96094  3729.40596 

4. 

657 

0.0001 

Error 

100  80079. 

90625  800.79906 

C 

Total 

127  180773. 

86719 

Root 

MSE 

28.29839 

R-square 

0.5570 

Dep  Mean 

145.67969 

Adj  R-aq 

0.4374 

C.V. 

19.42508 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

145.679688 

2.50124822 

58.243 

0.0001 

B1 

1 

-22.179688 

6.61768077 

-3.352 

0.0011 

B2 

1 

-0.054687 

6.61768077 

-0.008 

0.9934 

B3 

1 

14.507813 

6.61768077 

2.192 

0.0307 

B4 

1 

5.757813 

6.61768077 

0.870 

0.3863 

B5 

1 

14.507813 

6.61768077 

2.192 

0.0307 

B6 

1 

-7.804687 

6.61768077 

-1.179 

0.2410 

B7 

1 

-24.929688 

6.61768077 

-3.767 

0.0003 

B 

1 

15.335938 

2.50124822 

6.131 

0.0001 

AK 

1 

-6.914063 

2.50124822 

-2.764 

0.0068 

6 

1 

5.898438 

2.50124822 

2.358 

0.0203 

CE 

1 

-4.929688 

2.50124822 

-1.971 

0.0515 

E 

1 

4.898438 

2.50124822 

1.958 

0.0530 

CJ 

1 

4.351563 

2.50124822 

1.740 

0.0850 

EF 

1 

-4.304688 

2.50124822 

-1.721 

0.0883 

HK 

1 

-3.914063 

2.50124822 

-1.565 

0.1208 

BC 

1 

-3.804688 

2.50124822 

-1.521 

0.1314 

HJ 

1 

-3.460938 

2.50124822 

-1.384 

0.1695 

CH 

1 

3.273438 

2.50124822 

1.309 

0.1936 

D 

1 

-3.226563 

2.50124822 

-1.290 

0.2000 

AB 

1 

3.132813 

2.50124822 

1.252 

0.2133 

BH 

1 

3.039063 

2.50124822 

1.215 

0.2272 

AH 

1 

-3.007813 

2.50124822 

-1.203 

0.2320 

K 

1 

2.914063 

2.50124822 

1.165 

0.2468 

AC 

1 

-2.851563 

2.50124822 

-1.140 

0.2570 

GX 

1 

2.695313 

2.50124822 

1.078 

0.2838 

DE 

1 

2.617188 

2.50124822 

1.046 

0.2979 

AG 

1 

-2.617188 

2.50124822 

-1.046 

0.2979 

Attack  Caae  Max  Adj  Raqr  Day  9 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

31  103707. 

96875  3345. 

41835 

5 

.416 

0.0001 

Error 

96  59293. 

90625  617. 

64486 

C 

Total 

127  163001. 

87500 

Root 

MSE 

24.85246 

R- square 

0.6362 

Dep  Mean 

137.28125 

Adj  R-sq 

0.5188 

C.V. 

18.10332 

Parameter  Estimates 

Parameter 

Standard 

T 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T I 

INTERCEP 

1 

137.281250 

2.19666803 

62.495 

0.0001 

B1 

1 

-21.531250 

5.81183733 

-3.705 

0.0004 

B2 

1 

-0.531250 

5.81183733 

-0.091 

0.9274 

B3 

1 

17.531250 

5.81183733 

3.016 

0.0033 

B4 

1 

11.468750 

5.81183733 

1.973 

0.0513 

BS 

1 

16.156250 

5.81183733 

2.780 

0.0065 

B6 

1 

-11.656250 

5.81183733 

-2.006 

0.0477 

B7 

1 

-30.968750 

5.81183733 

-5.329 

0.0001 

B 

1 

12.156250 

2.19666803 

5.534 

0.0001 

6 

1 

9.093750 

2.19666803 

4.140 

0.0001 

E 

1 

6.765625 

2.19666803 

3.080 

0.0027 

AH 

1 

-4.234375 

2.19666803 

-1.928 

0.0569 

AE 

1 

-4.000000 

2.19666803 

-1.821 

0.0717 

AK 

1 

-3.937500 

2.19666803 

-1.792 

0.0762 

AG 

1 

-3.890625 

2.19666803 

-1.771 

0.0797 

BC 

1 

-3.640625 

2.19666803 

-1.657 

0.1007 

HK 

1 

-3.578125 

2.19666803 

-1.629 

0.1066 

FJ 

1 

-3.546875 

2.19666803 

-1.615 

0.1097 

H 

1 

3.343750 

2.19666803 

1.522 

0.1312 

CE 

1 

-2.937500 

2.19666803 

-1.337 

0.1843 

J 

1 

2.906250 

2.19666803 

1.323 

0.1890 

DJ 

1 

-2.890625 

2.19666803 

-1.316 

0.1913 

A 

1 

2.703125 

2.19666803 

1.231 

0.2215 

EH 

1 

2.703125 

2.19666803 

1.231 

0.2215 

K 

1 

2.671875 

2.19666803 

1.216 

0.2268 

FG 

1 

2.640625 

2.19666803 

1.202 

0.2323 

BH 

1 

2.593750 

2.19666803 

1.181 

0.2406 

CG 

1 

-2.484375 

2.19666803 

-1.131 

0.2609 

BF 

1 

-2.484375 

2.19666803 

-1.131 

0.2609 

AB 

1 

2.359375 

2.19666803 

1.074 

0.2855 

DG 

1 

-2.328125 

2.19666803 

-1.060 

0.2919 

EF 

1 

-2.312500 

2.19666803 

-1.053 

0.2951 

A-9 


Attack  Casa  Max  Adj  Rsqr  Day  10 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

27  105628. 

00000  3912.14815 

5. 

669 

0.0001 

Error 

100  69009. 

50000  690.09500 

C  Total 

127  174637. 

50000 

Root 

MSB 

26.26966 

R-square 

0.6048 

Dep  Mean 

132.31250 

Adj  R-sq 

0.4981 

C.V. 

19.85425 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob 

>  1 T  I 

INTERCEP 

1 

132.312500 

2.32193178 

56.984 

0.0001 

81 

1 

-20.000000 

6.14325405 

-3.256 

0.0015 

B2 

1 

2.437500 

6.14325405 

0.397 

0.6924 

B3 

1 

21.500000 

6.14325405 

3.500 

0.0007 

B4 

1 

4.312500 

6.14325405 

0.702 

0.4843 

BS 

1 

14.312500 

6.14325405 

2.330 

0.0218 

B6 

1 

-6.000000 

6.14325405 

-0.977 

0.3311 

B7 

1 

-32.125000 

6.14325405 

-5.229 

0.0001 

B 

1 

14.156250 

2.32193178 

6.097 

0.0001 

G 

1 

10.062500 

2.32193178 

4.334 

0.0001 

HK 

1 

-5.609375 

2.32193178 

-2.416 

0.0175 

BJ 

1 

5.078125 

2.32193178 

2.187 

0.0311 

E 

1 

4.562500 

2.32193178 

1.965 

0.0522 

FJ 

1 

-4.468750 

2.32193178 

-1.925 

0.0571 

AH 

1 

-3.609375 

2.32193178 

-1.554 

0.1232 

H 

1 

3.562500 

2.32193178 

1.534 

0.1281 

CJ 

1 

3.515625 

2.32193178 

1.514 

0.1332 

JK 

1 

-3.281250 

2.32193178 

-1.413 

0.1607 

AK 

1 

-3.218750 

2.32193178 

-1.386 

0.1688 

BG 

1 

3.093750 

2.32193178 

1.332 

0.1858 

CE 

1 

-3.093750 

2.32193178 

-1.332 

0.1858 

HJ 

1 

-2.796875 

2.32193178 

-1.205 

0.2312 

BC 

1 

-2.78125 0 

2.32193178 

-1.198 

0.2338 

DF 

1 

-2.703125 

2.32193178 

-1.164 

0.2471 

CD 

1 

2.531250 

2.32193178 

1.090 

0.2783 

BF 

1 

-2.453125 

2.32193178 

-1.057 

0.2933 

AD 

1 

2.453125 

2.32193178 

1.057 

0.2933 

CG 

1 

-2.375000 

2.32193178 

-1 . 023 

0.3088 
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Attack  Case  Max  Adj  Rsqr  Day  11 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

28  113769. 

56250  4063.19866 

7 

.160 

0.0001 

Error 

99  56182. 

43750  567.49937 

C 

Total 

127  169952. 

00000 

Root 

MSE 

23.82225 

R-square 

0.6694 

Dep  Mean 

127.12500 

Adj  R-sq 

0.5759 

C.V. 

18.73923 

Parameter  Estimates 

Parameter 

Standard  T  for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  |T| 

INTERCEP 

1 

127.125000 

2.10560889 

60.374 

0.0001 

B1 

1 

-16.250000 

5.57091749 

-2.917 

0.0044 

B2 

1 

-1.187500 

5.57091749 

-0.213 

0.8316 

B3 

1 

14.375000 

5.57091749 

2.580 

0.0113 

B4 

1 

1.625000 

5.57091749 

0.292 

0.7711 

B5 

1 

17.125000 

5.57091749 

3.074 

0.0027 

B6 

1 

-9.062500 

5.57091749 

-1.627 

0.1070 

B7 

1 

-30.187500 

5.57091749 

-5.419 

0.0001 

B 

1 

15.765625 

2.10560889 

7.487 

0.0001 

G 

1 

9.718750 

2.10560889 

4.616 

0.0001 

AK 

1 

-5.437500 

2.10560889 

-2.582 

0.0113 

JK 

1 

-5.203125 

2.10560889 

-2.471 

0.0152 

CJ 

1 

4.937500 

2.10560889 

2.345 

0.0210 

EF 

1 

-4.453125 

2.10560889 

-2.115 

0.0370 

HK 

1 

-3.890625 

2.10560889 

-1.848 

0.0676 

C 

1 

3.890625 

2.10560889 

1.848 

0.0676 

K 

1 

3.843750 

2.10560889 

1.825 

0.0709 

FK 

1 

3.781250 

2.10560889 

1.796 

0.0756 

AH 

1 

-3.609375 

2.10560889 

-1.714 

0.0896 

DG 

1 

-3.562500 

2.10560889 

-1.692 

0.0938 

E 

1 

3.390625 

2.10560889 

1.610 

0.1105 

H 

1 

3.328125 

2.10560889 

1.581 

0.1172 

DE 

1 

3.078125 

2.10560889 

1.462 

0.1469 

BK 

1 

2.765625 

2.10560889 

1.313 

0.1921 

BD 

1 

-2.703125 

2.10560889 

-1.284 

0.2022 

DH 

1 

2.640625 

2.10560889 

1.254 

0.2128 

CD 

1 

2.421875 

2.10560889 

1.150 

0.2528 

CF 

1 

2.328125 

2.10560889 

1.106 

0.2715 

AD 

1 

2.250000 

2.10560889 

1.069 

0.2879 

« 
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Attack  Case  Max  Adj  Rsqr  Day  12 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

26  113489. 

00000  4364. 

96154 

8. 

604 

0.0001 

Error 

101  51241. 

87500  507. 

34530 

C  Total 

127  164730. 

87500 

Root 

MSE 

22.52433 

R-square 

0.6889 

Dep  Mean 

121.15625 

Adj  R-aq 

0.6089 

C.V. 

18.59114 

Parameter  Estimates 

Parameter 

Standard 

T 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter~0 

Prob  >  | T I 

INTERCEP 

1 

121.156250 

1.99088803 

60.855 

0.0001 

B1 

1 

-16.906250 

5.26739461 

-3.210 

0.0018 

B2 

1 

-2.906250 

5.26739461 

-0.552 

0.5823 

B3 

1 

17.843750 

5.26739461 

3.388 

0.0010 

B4 

1 

2.031250 

5.26739461 

0.386 

0.7006 

B5 

1 

16.156250 

5.26739461 

3.067 

0.0028 

B6 

1 

-8.718750 

5.26739461 

-1.655 

0.1010 

B7 

1 

-33.093750 

5.26739461 

-6.283 

0.0001 

B 

1 

16.000000 

1.99088803 

8.037 

0.0001 

6 

1 

8.859375 

1.99088803 

4.450 

0.0001 

H 

1 

5.281250 

1.99088803 

2.653 

0.0093 

AK 

1 

-5.000000 

1.99088803 

-2.511 

0.0136 

HK 

1 

-4.515625 

1.99088803 

-2.268 

0.0254 

FK 

1 

4.359375 

X. 99088803 

2.190 

0.0309 

BH 

1 

4.062500 

1.99088803 

2.041 

0.0439 

EF 

1 

-3.828125 

1.99088803 

-1.923 

0.0573 

JK 

1 

-3.812500 

1.99088803 

-1.915 

0.0583 

E 

1 

3.359375 

1.99088803 

1.687 

0.0946 

C 

1 

3.250000 

1.99088803 

1.632 

0.1057 

BD 

1 

-3.062500 

1.99088803 

-1.538 

0.1271 

AH 

1 

-2.921875 

1.99088803 

-1.468 

0.1453 

DH 

1 

2.812500 

1.99088803 

1.413 

0.1608 

D 

1 

-2.718750 

1.99088803 

-1.366 

0.1751 

FH 

1 

2.593750 

1.99088803 

1.303 

0.1956 

GJ 

1 

2.250000 

1.99088803 

1.130 

0.2611 

A 

1 

2.171875 

1.99088803 

1.091 

0.2779 

CE 

1 

-2.140625 

1.99088803 

-1.075 

0.2848 
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Attack  Casa  Max  Adj  Raqr  Day  13 
Analysis  of  Variance 
Sum  of  Mean 


Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  109012. 

57031  4360.50281 

8 

.484 

0.0001 

Error 

102  52425. 

42188  513.97472 

C  Total 

127  161437. 

99219 

Root 

MSE 

22.67101 

R-square 

0.6753 

Dep  Mean 

116.49219 

Adj  R-sq 

0.5957 

C.V. 

19.46140 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

»  >  1 T  | 

INTERCEP 

1 

116.492188 

2.00385317 

58.134 

0.0001 

B1 

1 

>18.804688 

5.30169716 

-3.547 

0.0006 

B2 

1 

3.132812 

5.30169716 

0.591 

0.5559 

B3 

1 

15.632813 

5.30169716 

2.949 

0.0040 

B4 

1 

6.257813 

5.30169716 

1.180 

0.2406 

BS 

1 

13.007813 

5.30169716 

2.454 

0.0158 

B6 

1 

-4.179687 

5.30169716 

-0.788 

0.4323 

B7 

1 

-36.679687 

5.30169716 

-6.918 

0.0001 

B 

1 

16.820313 

2.00385317 

8.394 

0.0001 

G 

1 

9.726563 

2.00385317 

4.854 

0.0001 

JK 

1 

-4.367188 

2.00385317 

-2.179 

0.0316 

EJ 

1 

4.132813 

2.00385317 

2.062 

0.0417 

AK 

1 

-3.835938 

2.00385317 

-1.914 

0.0584 

BH 

1 

3.664063 

2.00385317 

1.829 

0.0704 

H 

1 

3.210938 

2.00385317 

1.602 

0.1122 

AH 

1 

-3.148438 

2.00385317 

-1.571 

0.1192 

BF 

1 

-3.132813 

2.00385317 

-1.563 

0.1211 

HK 

1 

-2.726563 

2.00385317 

-1.361 

0.1766 

C 

1 

2.554688 

2.00385317 

1.275 

0.2052 

FK 

1 

2.476563 

2.00385317 

1.236 

0.2193 

DH 

1 

2.351563 

2.00385317 

1.174 

0.2433 

EF 

1 

-2.335938 

2.00385317 

-1.166 

0.2464 

CG 

1 

-2.210938 

2.00385317 

-1.103 

0.2725 

AD 

1 

2.210938 

2.00385317 

1.103 

0.2725 

EG 

1 

-2.054688 

2.00385317 

-1.025 

0.3076 

GJ 

1 

1.960938 

2.00385317 

0.979 

0.3301 
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Attack  Casa  Max  Adj  Raqr  Day  14 
Analysis  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

30  110732. 

59375  3691.08646 

6. 

717 

0.0001 

Error 

C  Total 

Root  MSB 
Dep  Mean 
C.V. 

Variable  DF 

97  53302 . 

127  164035. 

23.44173 

110.31250 

21.25029 

Para 

Parameter 

Estimate 

90625  549.51450 

50000 

R-square  0.6751 

Adj  R-sq  0.5746 

meter  Estimates 

Standard  T  for  HO: 

Error  Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

110.312500 

2.07197539 

53.240 

0.0001 

B1 

1 

-18.125000 

5.48193160 

-3.306 

0.0013 

B2 

1 

-0.500000 

5.48193160 

-0.091 

0.9275 

B3 

1 

17.000000 

5.48193160 

3.101 

0.0025 

B4 

1 

10.250000 

5.48193160 

1.870 

0.0645 

B5 

1 

16.187500 

5.48193160 

2.953 

0.0039 

B6 

1 

-10.437500 

5.48193160 

-1.904 

0.0599 

B7 

1 

-37.312500 

5.48193160 

-6.806 

0.0001 

B 

1 

14.093750 

2.07197539 

6.802 

0.0001 

6 

1 

8.546875 

2.07197539 

4.125 

0.0001 

H 

1 

5.250000 

2.07197539 

2.534 

0.0129 

FK 

1 

5.156250 

2.07197539 

2.489 

0.0145 

AH 

1 

-4.265625 

2.07197539 

-2.059 

0.0422 

EJ 

1 

4.062500 

2.07197539 

1.961 

0.0528 

CG 

1 

-3.843750 

2.07197539 

-1.855 

0.0666 

DH 

1 

3.406250 

2.07197539 

1.644 

0.1034 

EF 

1 

-3.359375 

2.07197539 

-1.621 

0.1082 

HK 

1 

-3.218750 

2.07197539 

-1.553 

0.1236 

E 

1  -- 

3.078125 

2.07197539 

1.486 

0.1406 

JK 

1 

-2.859375 

2.07197539 

-1.380 

0.1708 

CD 

1 

-2.765625 

2.07197539 

-1.335 

0.1851 

AK 

1 

-2.640625 

2.07197539 

-1.274 

0.2055 

BH 

1 

2.593750 

2.07197539 

1.252 

0.2136 

AB 

1 

2.578125 

2.07197539 

1.244 

0.2164 

C 

1 

2.453125 

2.07197539 

1.184 

0.2393 

GJ 

1 

2.343750 

2.07197539 

1.131 

0.2608 

FJ 

1 

-2.265625 

2.07197539 

-1.093 

0.2769 

AG 

1 

-2.125000 

2.07197539 

-1.026 

0.3076 

CE 

1 

-2.125000 

2.07197539 

-1.026 

0.3076 

F 

1 

2.093750 

2.07197539 

1.011 

0.3148 

DG 

1 

-2.078125 

2.07197539 

-1.003 

0.3184 
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Attack  Case  Max  Adj  Raqr  Day  IS 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  108842. 

03125  4353. 

68125 

7 

.816 

0.0001 

Error 

102  56817. 

84375  557. 

03768 

C 

Total 

127  165659. 

87500 

Root 

MSE 

23.60165 

R-aquare 

0.6570 

Dep  Mean 

106.78125 

Adj  R-aq 

0.5730 

C.V. 

22.10280 

Parameter  Estimates 

Parameter 

Standard 

T  : 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

106.781250 

2.08611047 

51.187 

0.0001 

B1 

1 

-19.843750 

5.51932952 

-3.595 

0.0005 

B2 

1 

2.406250 

5.51932952 

0.436 

0.6638 

B3 

1 

12.031250 

5 . 519329S2 

2.180 

0.0316 

B4 

1 

12.781250 

5.51932952 

2.316 

0.0226 

B5 

1 

19.218750 

5.51932952 

3.482 

0.0007 

B6 

1 

-10.781250 

5.51932952 

-1.953 

0.0535 

B7 

1 

-34.031250 

5.51932952 

-6.166 

0.0001 

B 

1 

15.578125 

2.08611047 

7.468 

0.0001 

6 

1 

7.796875 

2.08611047 

3.738 

0.0003 

AK 

1 

-6.062500 

2.08611047 

-2.906 

0.0045 

EJ 

1 

5.015625 

2.08611047 

2.404 

0.0180 

H 

1 

4.609375 

2.08611047 

2.210 

0.0294 

C 

1 

4.265625 

2.08611047 

2.045 

0.0435 

JK 

1 

-4.234375 

2.08611047 

-2.030 

0.0450 

FK 

1 

3.953125 

2.08611047 

1.895 

0.0609 

AC 

1 

-3.578125 

2.08611047 

-1.715 

0.0893 

BH 

1 

3.500000 

2.08611047 

1.678 

0.0965 

AG 

1 

-3.421875 

2.08611047 

-1.640 

0.1040 

GK 

1 

3.3593’  5 

2.08611047 

1.610 

0.1104 

F 

1 

2.609375 

2.08611047 

1.251 

0.2139 

CF 

1 

2.437500 

2.08611047 

1.168 

0.2454 

AD 

1 

2.187500 

2.08611047 

1.049 

0.2968 

CE 

1 

-2.046875 

2.08611047 

-0.981 

0.3288 

E 

1 

1.937500 

2.08611047 

0.929 

0.3552 

EF 

1 

-1.734375 

2.08611047 

-0.831 

0.4077 

4 
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Attack  Case  Max  Adj  Rsqr  Day  16 


Analysis  of  Variance 


Source 

Sum  of 

DF  Squares 

Mean 

Square 

F  Value 

Prob>F 

Model 

Error 

C  Total 

33  97796.69531 

94  42620.10938 

127  140416.80469 

2963.53622 

453.40542 

6.536 

0.0001 

Root  MSE 
Dep  Mean 

C.V. 

21.29332  R-square 

100.03906  Adj  R-sq 

21.28500 

0.6965 

0.5899 

Parameter  Estimates 


Parameter  Standard  T  for  HO: 

Variable  DP  Estimate  Error  Parameter-0  Prob  >  |T| 

INTERCEP  1  100.039063  1.88208125 

B1  1  -22.101563  4.97951894 

1  0.023437  4.97951894 

1  11.023438  4.97951894 

1  10.898438  4.97951894 

1  18.523438  4.97951894 

1  -7.914063  4.97951894 

1  -30.914063  4.97951894 

1  13.429688  1.88208125 

1  7.695313  1.88208125 

1  4.742188  1.88208125 

1  4.554688  1.88208125 

1  -4.398438  1.88208125 

1  4.070313  1.88208125 

1  -3.992188  1.88208125 

1  -3.757813  1.88208125 

1  3.429688  1.88208125 

1  3.257813  1.88208125 

1  2.695313  1.88208125 

1  -2.664063  1.88208125 

1  2.539063  1.88208125 

1  -2.335938  1.88208125 

1  2.320313  1.88208125 

1  2.304688  1.88208125 

1  -2.195313  1.88208125 

1  -2.164063  1.88208125 

1  2.164063  1.88208125 

1  2.085938  1.88208125 

1  -2.070313  1.88208125 

1  -2.070313  1.88208125 

1  2.007813  1.88208125 

OH  1  1.945313  1.88208125 

CG  1  -1.945313  1.88208125 

F  1  1.898438  1.88208125 


53.153 

0.0001 

-4.438 

0.0001 

0.005 

0.9963 

2.214 

0.0293 

2.189 

0.0311 

3.720 

0.0003 

-1.589 

0.1153 

-6.208 

0.0001 

7.136 

0.0001 

4.089 

0.0001 

2.520 

0.0134 

2.420 

0.0174 

-2.337 

0.0216 

2.163 

0.0331 

-2.121 

0.0365 

-1.997 

0.0488 

1.822 

0.0716 

1.731 

0.0867 

1.432 

0.1554 

-1.415 

0.1602 

1.349 

0.1806 

-1.241 

0.2176 

1.233 

0.2207 

1.225 

0.2238 

-1.166 

0.2464 

-1.150 

0.2531 

1.150 

0.2531 

1.108 

0.2706 

-1.100 

0.2741 

-1.100 

0.2741 

1.067 

0.2888 

1.034 

0.3040 

-1.034 

0.3040 

1.009 

0.3157 
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Attack  Casa  Max  Adj  Rsqr  Day  17 


Analysis  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

26  97814. 

32813  3762.08954 

7 

.559 

0.0001 

Error 

C  Total 

Root  MSE 
Dep  Mean 

C.V. 

Variable  DF 

101  50267. 

127  148082. 

22.30922 

95.85156 

23.27476 

Para 

Parameter 

Estimate 

85156  497.70150 

17969 

R-square  0 . 6605 

Adj  R-sq  0.5732 

meter  Estimates 

Standard  T  for  HO: 

Error  Parameter*^ 

Prob  >  | T I 

INTERCEP 

1 

95.851563 

1.97187550 

48.609 

0.0001 

B1 

1 

-17.351563 

5.21709218 

-3.326 

0.0012 

B2 

1 

1.148437 

5.21709218 

0.220 

0.8262 

B3 

1 

10.023438 

5.21709218 

1.921 

0.0575 

B4 

1 

8.523438 

5.21709218 

1.634 

0.1054 

B5 

1 

18.960938 

5.21709218 

3.634 

0.0004 

B6 

1 

-5.539062 

5.21709218 

-1.062 

0.2909 

B7 

1 

-35.601562 

5.21709218 

-6.824 

0.0001 

B 

1 

12.710938 

1.97187550 

6.446 

0.0001 

G 

1 

6.820313 

1.97187550 

3.459 

0.0008 

AK 

1 

-5.507813 

1.97187550 

-2.793 

0.0062 

JK 

1 

-5.070313 

1.97187550 

-2.571 

0.0116 

FK 

1 

4.664063 

1.97187550 

2.365 

0.0199 

H 

1  . 

4.601563 

1.97187550 

2.334 

0.0216 

FG 

1 

-4.554688 

1.97187550 

-2.310 

0.0229 

EG 

1 

-4.351563 

1.97187550 

-2.207 

0.0296 

BK 

1 

3.929688 

1.97187550 

1.993 

0.0490 

HK 

1 

-3.929688 

1.97187550 

-1.993 

0.0490 

EJ 

1 

3.070313 

1.97187550 

1.557 

0.1226 

GK 

1 

3.070313 

1.97187550 

1.557 

0.1226 

BH 

1 

2.898438 

1.97187550 

1.470 

0.1447 

CE 

1 

-2.882813 

1.97187550 

-1.462 

0.1469 

CJ 

1 

2.867188 

1.97187550 

1.454 

0.1490 

J 

1 

2.710938 

1.97187550 

1.375 

0.1722 

C 

1 

2.601563 

1.97187550 

1.319 

0.1900 

CD 

1 

-2.570313 

1.97187550 

-1.303 

0.1954 

HJ 

1 

2.492188 

1.97187550 

1.264 

0.2092 

Attack  C Max  Adj  Raqr  Day  18 


Analysis  of  Varlancs 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Modal 

29  87353. 

00000  3012.17241 

6. 

498 

0.0001 

Error 

98  45424. 

87500  463.51913 

C  Total 

127  132777. 

87500 

Root 

MSE 

21.52949 

R-aquare 

0.6579 

Dap  Mean 

89.03125 

Adj  R-sq 

0.5567 

C.V. 

24.18195 

Para 

meter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

|  >  ITI 

INTERCEP 

1 

89.031250 

1.90295644 

46.786 

0.0001 

B1 

1 

-19.906250 

5.03474950 

-3.954 

0.0001 

B2 

1 

0.531250 

5.03474950 

0.106 

0.9162 

B3 

1 

6.656250 

5.03474950 

1.322 

0.1892 

B4 

1 

4.656250 

5.03474950 

0.925 

0.3573 

B5 

1 

14.968750 

5.03474950 

2.973 

0.0037 

B6 

1 

-6.093750 

5.03474950 

-1.210 

0.2291 

B7 

1 

-26.281250 

5.03474950 

-5.220 

0.0001 

B 

1 

11.187500 

1.90295644 

5.879 

0.0001 

G 

1 

7.328125 

1.90295644 

3.851 

0.0002 

H 

1 

5.156250 

1.90295644 

2.710 

0.0080 

GK 

1 

5.156250 

1.90295644 

2.710 

0.0080 

EJ 

1 

5.140625 

1.90295644 

2.701 

0.0081 

HK 

1 

-5.015625 

1.90295644 

-2 . 636 

0.0098 

CD 

1 

-4.328125 

1.90295644 

-2.274 

0.0251 

JK 

1 

-4.031250 

1.90295644 

-2.118 

0.0367 

J 

1 

3.734375 

1.90295644 

1.962 

0.0526 

FK 

1 

3.078125 

1.90295644 

1.618 

0.1090 

AK 

1 

-3.046875 

1.90295644 

-1.601 

0.1126 

EF 

1 

-3.031250 

1.90295644 

-1.593 

0.1144 

FH 

1 

2.968750 

1.90295644 

1.560 

0.1220 

CE 

1 

-2.828125 

1.90295644 

-1.486 

0.1404 

GJ 

1 

2.656250 

1.90295644 

1.396 

0.1659 

BH 

1 

2.562500 

1.90295644 

1.347 

0.1812 

AE 

1 

2.531250 

1.90295644 

1.330 

0.1866 

C 

1 

2.453125 

1.90295644 

1.289 

0.2004 

EG 

1 

-2.453125 

1.90295644 

-1.289 

0.2004 

CF 

1 

2.265625 

1.90295644 

1.191 

0.2367 

FG 

1 

-2.265625 

1.90295644 

-1.191 

0.2367 

BE 

1 

-2.218750 

1.90295644 

-1.166 

0.2465 

A- 18 


Attack  Case  Max  Ad j  Raqr  Day  19 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

30  88597. 

48437  2953.24948 

5. 

994 

0.0001 

Error 

97  47792. 

44531  492.70562 

C 

Total 

127  136389. 

92969 

Root 

MSE 

22.19697 

R-square 

0.6496 

Dep  Mean 

86.52344 

Adj  R-sq 

0.5412 

C.V. 

25.65429 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  1 T | 

INTERCEP 

1 

86.523438 

1.96195379 

44.101 

0.0001 

B1 

1 

-19.210938 

5.19084181 

-3.701 

0.0004 

B2 

1 

-2.085938 

5.19084181 

-0.402 

0.6887 

B3 

1 

10.289063 

5.19084181 

1.982 

0.0503 

B4 

1 

7.914063 

5.19084181 

1.525 

0.1306 

B5 

1 

17.164063 

5.19084181 

3.307 

0.0013 

B6 

1 

-5.648437 

5.19084181 

-1.088 

0.2792 

B7 

1 

-29.085938 

5.19084181 

-5.603 

0.0001 

B 

1 

12.429688 

1.96195379 

6.335 

0.0001 

G 

1 

5.992188 

1.96195379 

3.054 

0.0029 

JK 

1 

-5.164063 

1.96195379 

-2 . 632 

0.0099 

H 

1 

5.085938 

1.96195379 

2.592 

0.0110 

BH 

1 

4.898438 

1.96195379 

2.497 

0.0142 

EJ 

1 

4.710938 

1.96195379 

2.401 

0.0182 

FK 

1 

4.398438 

1.96195379 

2.242 

0.0272 

J 

1 

4.117188 

1.96195379 

2.099 

0.0385 

FG 

1 

-3.320313 

1.96195379 

-1.692 

0.0938 

GK 

1 

3.304688 

1.96195379 

1.684 

0.0953 

HK 

1 

-3.195313 

1.96195379 

-1.629 

0.1066 

DH 

1 

2.914063 

1.96195379 

1.485 

0.1407 

CE 

1 

-2.820313 

1.96195379 

-1.438 

0.1538 

FH 

1 

2.804688 

1.96195379 

1.430 

0.1561 

HJ 

1 

2.804688 

1.96195379 

1.430 

0.1561 

AK 

1 

-2.257813 

1.96195379 

-1.151 

0.2526 

CD 

1 

-2.148438 

1.96195379 

-1.095 

0.2762 

DG 

1 

-2.117188 

1.96195379 

-1.079 

0.2832 

DK 

1 

2.070313 

1.96195379 

1.055 

0.2939 

FJ 

1 

-2.039063 

1.96195379 

-1.039 

0.3012 

F 

1 

2.023438 

1.96195379 

1.031 

0.3049 

CH 

1 

-2.007813 

1.96195379 

-1.023 

0.3087 

AE 

1 

1.992188 

1.96195379 

1.015 

0.3124 

Attack  Casa  Max  Adj  Rsqr  Day  20 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

OF  Squares 

Square 

F  Value 

Model 

30  84977.31250 

2832.57708 

7.243 

Error 

97  37936.90625 

391.10213 

C  Total 

127  122914.21875 

Root  MSE 

19.77630  R- 

■square 

0.6914 

Dep  Mean 

79.17188  Adj  R-aq 

0.5959 

C.V. 

24.97895 

Parameter  Estimates 


Variable  OF 


ZNTERCEP  1 
B1  1 

B2  1 

B3  1 

B4  1 

B5  1 

B6  1 

B7  1 

B  1 

G  1 

JK  1 

FK  1 

EJ  1 

BK  1 

AK  1 

BH  1 

GK  1 

H  1 

FH  1 

CE  1 

FG  1 

J  1 

EG  1 

F  1 

C  1 

CG  1 

HK  1 

CF  1 

BE  1 

HJ  1 

FJ  1 


Parameter 

Estimate 

79.171875 

-20.921875 

-0.921875 

11.078125 

7.578125 

14.953125 

-1.734375 

-27.921875 

11.265625 

7.109375 

-5.062500 

4.750000 

4.671875 

4.312500 

-4.109375 

4.031250 

3.718750 

3.562500 

3.531250 

-3.515625 

-3.203125 

2.953125 

-2.796875 

2.546875 

2.484375 

2.453125 

-2.390625 

2.328125 

-2.265625 

2.093750 

-2.015625 


Standard 

Error 

1.74799467 

4.62475919 

4.62475919 

4.62475919 

4.62475919 

4.62475919 

4.62475919 

4.62475919 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 

1.74799467 


T  for  HO: 

Parameters 

45.293 

-4.524 

-0.199 

2.395 

1.639 

3.233 

-0.375 

-6.037 

6.445 

4.067 

-2.896 

2.717 

2.673 

2.467 

-2.351 

2.306 

2.127 

2.038 

2.020 

-2.011 

-1.832 

1.689 

-1.600 

1.457 

1.421 

1.403 

-1.368 

1.332 

-1.296 

1.198 

-1.153 


Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


Prob>F 

0.0001 


>  ITI 

.0001 

.0001 

.8424 

.0185 

.1045 

.0017 

.7085 

.0001 

.0001 

.0001 

.0047 

.0078 

.0088 

.0154 

.0208 

.0232 

.0359 

.0443 

.0461 

.0471 

.0700 

.0943 

.1128 

.1483 

.1584 

.1637 

.1746 

.I860 

.1980 

.2339 

.2517 
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Attack  Case  Max  Adj  Raqr  Day  21 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

34  8S198. 

57813  2505.84053 

7 

.294 

0.0001 

Error 

93  31949. 

35156  343.54141 

C 

Total 

127  117147. 

92969 

Root 

MSE 

18.53487 

R-square 

0.7273 

Dep  Mean 

74.47656 

Adj  R-sq 

0.6276 

C.V. 

24.88685 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  | T I 

INTERCEP 

1 

74.476563 

1.63826655 

45.461 

0.0001 

B1 

1 

-20.476563 

4.33444588 

-4.724 

0.0001 

B2 

1 

1.335937 

4.33444588 

0.308 

0.7586 

B3 

1 

11.398438 

4.33444588 

2.630 

0.0100 

B4 

1 

9.023438 

4.33444588 

2.082 

0.0401 

B5 

1 

12.710938 

4.33444588 

2.933 

0.0042 

B6 

1 

-4.539063 

4.33444588 

-1.047 

0.2977 

B7 

1 

-29.351563 

4.33444588 

-6.772 

0.0001 

B 

1 

10.320313 

1.63826655 

6.300 

0.0001 

G 

1 

6.132813 

1.63826655 

3.743 

0.0003 

H 

1 

5.789063 

1.63826655 

3.534 

0.0006 

GK 

1 

5.523438 

1.63826655 

3.372 

0.0011 

EJ 

1 

4.398438 

1.63826655 

2.685 

0.0086 

FK 

1 

4.179688 

1.63826655 

2.551 

0.0124 

AK 

1 

-4.023438 

1.63826655 

-2.456 

0.0159 

JK 

1 

-3.851563 

1.63826655 

-2.351 

0.0208 

BE 

1 

-3.726563 

1.63826655 

-2.275 

0.0252 

BH 

1 

3.726563 

1.63826655 

2.275 

0.0252 

C 

1 

3.601563 

1.63826655 

2.198 

0.0304 

FH 

1 

3.476563 

1.63826655 

2.122 

0.0365 

CE 

1 

-2.914063 

1.63826655 

-1.779 

0.0785 

CG 

1 

2.820313 

1.63826655 

1.722 

0.0885 

F 

1 

2.601563 

1.63826655 

1.588 

0.1157 

EG 

1 

-2.570313 

1.63826655 

-1.569 

0.1201 

J 

1 

2.414063 

1.63826655 

1.474 

0.1440 

FG 

1 

-2.273438 

1.63826655 

-1.388 

0.1685 

FJ 

1 

-2.273438 

1.63826655 

-1.388 

0.1685 

DG 

1 

-2.164063 

1.63826655 

-1.321 

0.1898 

A 

1 

-2.164063 

1.63826655 

-1.321 

0.1898 

GJ 

1 

2.101563 

1.63826655 

1.283 

0.2028 

DH 

1 

2.085938 

1.63826655 

1.273 

0.2061 

HK 

1 

-1.945313 

1.63826655 

-1.187 

0.2381 

AE 

1 

1.882813 

1.63826655 

1.149 

0.2534 

E 

1 

-1.789063 

1.63826655 

-1.092 

0.2776 

AJ 

1 

1.773438 

1.63826655 

1.083 

0.2818 

A-21 


S"888gaB3gc,Sgga0SaS3aS*°“5£SS8SS| 


Attack  Casa  Max  Adj  Raqr  Day  22 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares 

Square 

F  Value 

Model 

34  67651.37500 

1989.74632 

6.729 

Error 

93  27500.62500 

295.70565 

C  Total 

127  95152.00000 

Root  MSE 

17.19609  R-square 

0.7110 

Dep  Mean 

69.62500  Adj  R-sq 

0.6053 

C.V. 

24.69816 

Parameter  Estimates 


Variable 
ERCEP 


DH 

F 


Parameter 

Standard 

T  for  HO: 

DF 

Estimate 

Error 

Paramater-0 

1 

69.625000 

1.51993433 

45.808 

1 

-18.000000 

4.02136823 

-4.476 

1 

-1.562500 

4.02136823 

-0.389 

1 

11.125000 

4.02136823 

2.766 

1 

10.500000 

4.02136823 

2.611 

1 

8.312500 

4.02136823 

2.067 

1 

-3.312500 

4.02136823 

-0.824 

1 

-24.812500 

4.02136823 

-6.170 

1 

8.937500 

1.S1993433 

5.880 

1 

7.718750 

1.51993433 

5.078 

1 

4.765625 

1.51993433 

3.135 

1 

4.406250 

1.51993433 

2.899 

1 

-4.203125 

1.51993433 

-2.765 

1 

3.750000 

1.51993433 

2.467 

1 

3.437500 

1.51993433 

2.262 

1 

-3.312500 

1.51993433 

-2.179 

1 

3.015625 

1.51993433 

1.984 

1 

3.015625 

1.51993433 

1.984 

1 

-2.765625 

1.51993433 

-1.820 

1 

-2.750000 

1.51993433 

-1.809 

1 

2.671875 

1.51993433 

1.758 

1 

2.562500 

1.51993433 

1.686 

1 

2.343750 

1.51993433 

1.542 

1 

-2.281250 

1.51993433 

-1.501 

1 

-2.218750 

1.51993433 

-1.460 

1 

-2.093750 

1.51993433 

-1.378 

1 

-2.046875 

1.51993433 

-1.347 

1 

-2.031250 

1.51993433 

-1.336 

1 

-1.859375 

1.51993433 

-1.223 

1 

1.828125 

1.51993433 

1.203 

1 

1.734375 

1.51993433 

1.141 

1 

-1.718750 

1.51993433 

-1.131 

1 

1.703125 

1.51993433 

1.121 

1 

1.578125 

1.51993433 

1.038 

1 

1.562500 

1.51993433 

1.028 

Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

c 

0 

c 

c 

0 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


Prob>F 

0.0001 


>  |T| 

.0001 

.0001 

.6985 

.0068 

.0105 

.0415 

.4122 

.0001 

.0001 

.0001 

.0023 

.0047 

.0069 

.0154 

.0261 

.0318 

.0502 

.0502 

.0720 

.0736 

.0821 

.0952 

.1265 

.1368 

.1477 

1.1717 

1.1814 

(.1847 

>.2243 

>.2321 

1.2568 

1.2610 

1.2654 

>.3018 

>.3066 
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Attack  Case  Max  Adj  Rsqr  Day  23 


Analysis  of  Variance 


Sum  of 

Mean 

* 

Source 

OF 

Squares 

Square 

F  Value 

Model 

32 

64739.06250 

2023.09570 

5.834 

Error 

95 

32943.42969 

346.77294 

C  Total 

127 

97682.49219 

Root  MSB  18.62184  R-square  0.6627 
Dep  Mean  65.74219  Adj  R-sq  0.5491 
C.V.  28.32556 


Parameter  Estimates 


* 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Paramater-0 

Prob 

INTERCEP 

1 

65. 742188 

1.64595371 

39.942 

0 

B1 

1 

-21.054688 

4.35478419 

-4.835 

0 

B2 

1 

-0.554688 

4.35478419 

-0.127 

0 

B3 

1 

10.695313 

4.35478419 

2.456 

0 

B4 

1 

7.320313 

4.35478419 

1.681 

0 

B5 

1 

12.570313 

4.35478419 

2.887 

0 

B6 

1 

-4.054688 

4.35478419 

-0.931 

0 

B7 

1 

-22.867188 

4.35478419 

-5.251 

0 

B 

1 

9.570313 

1.64595371 

5.814 

0 

G 

1 

6.179688 

1.64595371 

3.754 

0 

H 

1 

4.804688 

1.64595371 

2.919 

0 

GK 

1 

3.929688 

1.64595371 

2.387 

0 

EJ 

1 

3.648438 

1.64595371 

2.217 

0 

FG 

1  ' 

-3.670313 

1.64595371 

-2.169 

0 

HK 

1 

-3.351563 

1.64595371 

-2.036 

0 

AK 

1 

-3.335938 

1.64595371 

-2.027 

0 

FK 

1 

3.242188 

1.64595371 

1.970 

0 

BE 

1 

-2.632813 

1.64595371 

-1.600 

0 

BH 

1 

2.539063 

1.64595371 

1.543 

0 

EG 

1 

-2.523438 

1.64595371 

-1.533 

0 

CK 

1 

2.460938 

1.64595371 

1.495 

0 

FH 

1 

2.460938 

1.64595371 

1.495 

0 

CG 

1 

2.398438 

1.64595371 

1.457 

0 

BK 

1 

2.289063 

1.64595371 

1.391 

0 

CE 

1 

-2.210938 

1.64595371 

-1.343 

0 

F 

1 

2.210938 

1.64595371 

1.343 

0 

C 

1 

2.148438 

1.64595371 

1.305 

0 

JK 

1 

-1.804688 

1.64595371 

-1.096 

0 

DH 

1 

1.726563 

1.64595371 

1.049 

0 

CH 

1 

-1.695313 

1.64595371 

-1.030 

0 

CD 

1 

-1.679688 

1.64595371 

-1.020 

0 

BF 

1 

1.664063 

1.64595371 

1.011 

0 

DG 

1 

-1.648438 

1.64595371 

-1.002 

0 

* 


Prob>F 

0.0001 


>  ITI 

.0001 

.0001 

.8989 

.0159 

.0961 

.0048 

.3542 

.0001 

.0001 

.0003 

.0044 

.0189 

.0290 

.0326 

.0445 

.0455 

.0518 

.1130 

.1263 

.1286 

.1382 

.1382 

.1484 

.1676 

.1824 

.1824 

.1949 

.2757 

.2969 

.3056 

.3101 

.3146 

.3191 
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Attack  Casa  Max  Adj  Rsqr  Day  24 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

29  69350. 

00000  2391.37931 

8. 

123 

0.0001 

Error 

96  28849. 

46875  294.38233 

C  Total 

127  98199. 

46875 

Root 

MSE 

17.15757 

R-square 

0.7062 

Dep  Mean 

62.14063 

Adj  R-sq 

0.6193 

C.V. 

27.61088 

Para 

meter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

•  >  IT  | 

INTERCEP 

1 

62.140625 

1.51652959 

40.976 

0.0001 

B1 

1 

-18.703125 

4.01236014 

-4.661 

0.0001 

B2 

1 

-6.203125 

4.01236014 

-1.546 

0.1253 

B3 

1 

17.109375 

4.01236014 

4.264 

0.0001 

B4 

1 

8.671875 

4.01236014 

2.161 

0.0331 

BS 

1 

10.046875 

4.01236014 

2.504 

0.0139 

B6 

1 

-4.328125 

4.01236014 

-1.079 

0.2834 

B7 

1 

-23.515625 

4.01236014 

-5.861 

0.0001 

B 

1 

10.406250 

1.51652959 

6.862 

0.0001 

6 

1 

5.500000 

1 . 51652959 

3.627 

0.0005 

H 

1 

5.218750 

1.51652959 

3.441 

0.0009 

BH 

1 

4.609375 

1.51652959 

3.039 

0.0030 

GK 

1 

4.000000 

1 . 51652959 

2.638 

0.0097 

EJ 

1 

3.843750 

1.51652959 

2.535 

0.0128 

FK 

1 

3.640625 

1.51652959 

2.401 

0.0183 

BK 

1 

3.375000 

1.51652959 

2.225 

0.0283 

CE 

1 

-3.109375 

1.51652959 

-2 . 050 

0.0430 

EG 

1 

-2.984375 

1.51652959 

-1.968 

0.0519 

F 

1 

2.765625 

1.51652959 

1.824 

0.0713 

FH 

1 

2.750000 

1.51652959 

1.813 

0.0728 

DG 

1 

-2.718750 

1.51652959 

-1.793 

0.0761 

AK 

1 

-2 . 609375 

1.51652959 

-1.721 

0.0885 

J 

1 

2.515625 

1.51652959 

1.659 

0.1004 

HK 

1 

-2.468750 

1.51652959 

-1.628 

0.1068 

FG 

1 

-2.437500 

1.51652959 

-1.607 

0.1112 

JK 

1 

-2.265625 

1.51652959 

-1.494 

0.1384 

CK 

1 

2.250000 

1.51652959 

1.484 

0.1411 

C 

1 

1.781250 

1.51652959 

1.175 

0.2430 

DH 

1 

1.656250 

1.51652959 

1.092 

0.2775 

CG 

1 

1.578125 

1.51652959 

1.041 

0.3006 
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Attack  Case  Max  Adj  Rsqr  Day 

25 

Analysis  of  Variance 

Sum  of  Mean 

• 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

33  67907. 

13281  2057.79190 

7 

.321 

0.0001 

Error 

94  26421. 

67188  281.08162 

C  Total 

127  94328. 

80469 

« 

Root 

MSE 

16.76549 

R-aquare 

0.7199 

Dep  Mean 

58.96094 

Adj  R-sq 

0.6216 

C.V. 

28.43491 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

>  1  T  | 

INTERCEP 

1 

58.960938 

1.48187386 

39.788 

0.0001 

B1 

1 

-17.335938 

3.92066970 

-4.422 

0.0001 

B2 

1 

-4.148438 

3.92066970 

-1.058 

0.2927 

B3 

1 

11.289063 

3.92066970 

2.879 

0.0049 

B4 

1 

11.601563 

3.92066970 

2.959 

0.0039 

BS 

1 

10.976563 

3.92066970 

2.800 

0.0062 

B6 

1 

-4.273438 

3.92066970 

-1.090 

0.2785 

B7 

1 

-24.710938 

3.92066970 

-6.303 

0.0001 

B 

1 

9.945313 

1.48187386 

6.711 

0.0001 

FK 

1 

5.195313 

1.48187386 

3.506 

0.0007 

H 

1 

5.132813 

1.48187386 

3.464 

0.0008 

EJ 

1 

4.585938 

1.48187386 

3.095 

0.0026 

G 

1 

4.492188 

1.48187386  * 

3.031 

0.0031 

BH 

1 

4.398438 

1.48187386 

2.968 

0.0038 

GK 

1 

4.101563 

1.48187386 

2.768 

0.0068 

HK 

1 

-3.695313 

1.48187386 

-2.494 

0.0144 

JK 

1 

-3.601563 

1.48187386 

-2.430 

0.0170 

EG 

1 

-2.585938 

1.48187386 

-1.745 

0.0842 

J 

1 

2.570313 

1.48187386 

1.735 

0.0861 

FG 

1 

-2.476563 

1.48187386 

-1.671 

0.0980 

CE 

1 

-2.460938 

1.48187386 

-1.661 

0.1001 

C 

1 

2.398438 

1.48187386 

1.619 

0.1089 

F 

1 

2.273438 

1.48187386 

1.534 

0.1283 

AK 

1 

-2.179688 

1.48187386 

-1.471 

0.1447 

GH 

1 

-2.117188 

1.48187386 

-1.429 

0.1564 

BG 

1 

-1.992188 

1.48187386 

-1.344 

0.1821 

EF 

1 

-1.929688 

1.48187386 

-1.302 

0.1960 

BE 

1 

-1.914063 

1.48187386 

-1.292 

0.1996 

FH 

1 

1.726563 

1.48187386 

1.165 

0.2469 

DE 

1 

1.710938 

1.48187386 

1.155 

0.2512 

AG 

1 

1.679688 

1.48187386 

1.133 

0.2599 

BK 

1 

1.679688 

1.48187386 

1.133 

0.2599 

DG 

1 

-1.617188 

1.48187386 

-1.091 

0.2779 

<r 

# 

DH 

1 

1.585938 

1.48187386 

1.070 

0.2873 
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Attack  Case  Max  Adj  Rsqr  Day  26 


Analysis  of 

Variance 

Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Prob>F 

Modal 

30 

59810.46875 

1993.68229 

7.108 

0.0001 

Error 

97 

27206.40625 

280.47841 

C  Total 

127 

87016.87500 

Root 

MSE 

16.74749 

R-square 

0.6873 

Dep  Mean 

55.59375 

Adj  R-sq 

0.5906 

C.V. 

30.12477 

Parameter  Estimates 

Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameters 

Prob  >  | T | 

INTERCEP 

1 

55.593750 

1.48028295 

37.556 

0.0001 

B1 

1 

-19.656250 

3.91646056 

-5.019 

0.0001 

B2 

1 

-0.968750 

3.91646056 

-0.247 

0.8052 

B3 

1 

12.531250 

3.91646056 

3.200 

0.0019 

B4 

1 

9.718750 

3.91646056 

2.482 

0.0148 

B5 

1 

12.031250 

3.91646056 

3.072 

0.0028 

B6 

1 

-2.531250 

3.91646056 

-0.646 

0.5196 

B7 

1 

-21.968750 

3.91646056 

-5.609 

0.0001 

B 

1 

8.750000 

1.48028295 

5.911 

0.0001 

6 

1 

5.546875 

1.48028295 

3.747 

0.0003 

BH 

1 

5.156250 

1.48028295 

3.483 

0.0007 

H 

1 

4.781250 

1.48028295 

3.230 

0.0017 

EJ 

1 

4.515625 

1.48028295 

3.051 

0.0029 

FK 

1 

4.406250 

1.48028295 

2.977 

0.0037 

GK 

1 

3.640625 

1.48028295 

2.459 

0.0157 

JK 

1 

-3.578125 

1.48028295 

-2.417 

0.0175 

CE 

1 

-3.312500 

1.48028295 

-2.238 

0.0275 

HK 

1 

-2.812500 

1.48028295 

-1.900 

0.0604 

EG 

1 

-2.578125 

1.48028295 

-1.742 

0.0847 

C 

1 

2.437500 

1.48028295 

1.647 

0.1029 

F 

1 

2.218750 

1.48028295 

1.499 

0.1372 

DE 

1 

2.187500 

1.48028295 

1.478 

0.1427 

DG 

1 

-2.078125 

1.48028295 

-1.404 

0.1636 

BK 

1 

2.031250 

1.48028295 

1.372 

0.1732 

EF 

1 

-1.937500 

1.48028295 

-1.309 

0.1937 

BJ 

1 

-1.828125 

1.48028295 

-1.235 

0.2198 

EK 

1 

-1.718750 

1.48028295 

-1.161 

0.2485 

AG 

1 

1.703125 

1.48028295 

1.151 

0.2527 

GH 

1 

-1.703125 

1.48028295 

-1.151 

0.2527 

BG 

1 

-1.640625 

1.48028295 

-1.108 

0.2705 

FG 

1 

-1.640625 

1.48028295 

-1.108 

0.2705 

A-26 


Attack  Casa  Max  Adj  Raqr  Day  27 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  ValCie 

Prob>F 

Model 

33  59023. 

87500  1788. 

60227 

8 

.582 

0.0001 

Error 

94  19591. 

62500  2°8 . 

42154 

C 

Total 

127  78615. 

50000 

Root 

MSE 

14.43681 

R- aqua re 

0.7508 

Dep  Mean 

51.56250 

Adj  R-aq 

0.6633 

C.V. 

27 . 99867 

Parameter  Estimates 

Parameter 

Standard 

T  : 

Cor  HO: 

Variable  DF 

Estimate 

Error 

Parameters 

Prob  >  IT | 

INTERCEP 

1 

51.562500 

1.27604596 

40.408 

0.0001 

B1 

1 

-19.250000 

3.37610028 

-5.702 

0.0001 

B2 

1 

-1.437500 

3.37610028 

-0.426 

0.6712 

B3 

1 

12.812500 

3.37610028 

3.795 

0.0003 

B4 

1 

10.125000 

3.37610028 

2.999 

0.0035 

B5 

1 

7.625000 

3.37610028 

2.259 

0.0262 

B6 

1 

1.125000 

3.37610028 

0.333 

0.7397 

B7 

1 

-22.187500 

3.37610028 

-6.572 

0.0001 

B 

1 

9.562500 

1.27604596 

7.494 

0.0001 

G 

1 

5.625000 

1.27604596 

4.408 

0.0001 

H 

1 

4.718750 

1.27604596 

3.698 

0.0004 

EJ 

1 

4.656250 

1.27604596 

3.649 

0.0004 

FK 

1 

4.578125 

1.27604596 

3.588 

0.0005 

BH 

1 

3.843750 

1.27604596 

3.012 

0.0033 

CE 

1 

-3.421875 

1.27604596 

-2 . 682 

0.0087 

GK 

1 

3.109375 

1.27604596 

2.437 

0.0167 

JK 

1 

-3.031250 

1.27604596 

-2.376 

0.0196 

HK 

1 

-2.796875 

1.27604596 

-2.192 

0.0309 

F 

1 

2.531250 

1.27604596 

1.984 

0.0502 

C 

1 

2.468750 

1.27604596 

1.935 

0.0560 

BK 

1 

2.390625 

1.27604596 

1.873 

0.0641 

EG 

1 

-2.265625 

1.27604596 

-1.776 

0.0791 

AK 

1 

-2.171875 

1.27604596 

-1.702 

0.0921 

DB 

1 

1.921875 

1.27604596 

1.506 

0.1354 

EH 

1 

1.828125 

1.27604596 

1.433 

0.1553 

AG 

1 

1.750000 

1.27604596 

1.371 

0.1735 

J 

1 

1.609375 

1.27604596 

1.261 

0.2104 

BG 

1 

-1.562500 

1.27604596 

-1.224 

0.2238 

DG 

1 

-1.546875 

1.27604596 

-1.212 

0.2285 

FG 

1 

-1.468750 

1.27604596 

-1.151 

0.2526 

FJ 

1 

-1.453125 

1.27604596 

-1.139 

0.2577 

A 

1 

-1.437500 

1.27604596 

-1.127 

0.2628 

GH 

1 

-1.343750 

1.27604596 

-1.053 

0.2950 

EK 

1 

-1.312500 

1.27604596 

-1.029 

0.3063 

Attack  Casa  Max  Adj  Raqr  Day  28 
Analyaia  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

27  51507. 

59375  1907.68866 

8 

.073 

0.0001 

Error 

100  23630. 

28125  236.30281 

C  Total 

127  75137. 

87500 

Root 

MSE 

15.37214 

R-aquare 

0.6855 

Dep  Mean 

48.71875 

Adj  R-aq 

0.6006 

C.V. 

31.55283 

Para 

meter  Eatimatea 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Eatimate 

Error  Parameter-0 

Proto 

-  >  1 T I 

INTERCEP 

1 

48.718750 

1.35871841 

35.856 

0.0001 

B1 

1 

-18.968750 

3.59483102 

-5.277 

0.0001 

B2 

1 

-0.656250 

3.59483102 

-0.183 

0.8555 

B3 

1 

10.343750 

3.59483102 

2.877 

0.0049 

B4 

1 

6.531250 

3.59483102 

1.817 

0.0722 

B5 

1 

7.343750 

3.59483102 

2.043 

0.0437 

B6 

1 

2.031250 

3.59483102 

0.565 

0.5733 

B7 

1 

-21.718750 

3.59483102 

-6.042 

0.0001 

B 

1 

8.640625 

1.35871841 

6.359 

0.0001 

FK 

1 

4.859375 

1.35871841 

3.576 

0.0005 

H 

1 

4.843750 

1.35871841 

3.565 

0.0006 

EJ 

1 

4.796875 

1.35871841 

3.530 

0.0006 

G 

1 

3.859375 

1.35871841 

2.840 

0.0055 

JK 

1 

-3.843750 

1.35871841 

-2.829 

0.0056 

GK 

1 

3.265625 

1.35871841 

2.403 

0.0181 

CE 

1 

-3.031250 

1.35871841 

-2.231 

0.0279 

HK 

1 

-3.000000 

1.35871841 

-2.208 

0.0295 

F 

1 

2 . 640625 

1.35871841 

1.943 

0.0548 

BG 

1 

-2.437500 

1.35871841 

-1.794 

0.0758 

FH 

1 

2.046875 

1.35871841 

1.506 

0.1351 

FG 

1 

-1.968750 

1.35871841 

-1.449 

0.1505 

BH 

1 

1.890625 

1.35871841 

1.391 

0.1672 

AG 

1 

1.750000 

1.35871841 

1.288 

0.2007 

EG 

1 

-1.718750 

1.35871841 

-1.265 

0.2088 

DH 

1 

1.671875 

1.35871841 

1.230 

0.2214 

AK 

1 

-1.640625 

1.35871841 

-1.207 

0.2301 

C 

1 

1.546875 

1.35871841 

1.138 

0.2576 

EF 

1 

-1.500000 

1.35871841 

-1.104 

0.2723 

A-28 


Attack  Casa  Max  Ad j  Rsqr  Day  29 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares 

Square 

F  Value 

Prob>F 

Model 

33  43646.25781 

1322.61387 

6.146 

0.0001 

Error 

94  20228.79688 

215.19997 

C  Total 

127  63875.05469 

Root  MSB 

14.66970  R- 

■square 

0.6833 

Dep  Mean 

46.58594  Adj  R-sq 

0.5721 

C.V. 

31.48954 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

46.585938 

1.29663015 

35.928 

0.0001 

B1 

1 

-17.398438 

3.43056091 

-5.072 

0.0001 

B2 

1 

-1.023438 

3.43056091 

-0.298 

0.7661 

B3 

1 

8.414063 

3.43056091 

2.453 

0.0160 

B4 

1 

9.664063 

3.43056091 

2.817 

0.0059 

B5 

1 

7.976563 

3.43056091 

2.325 

0.0222 

B6 

1 

0.789062 

3.43056091 

0.230 

0.8186 

B7 

1 

-20.085938 

3.43056091 

-5.855 

0.0001 

B 

1 

7.804688 

1.29663015 

6.019 

0.0001 

EJ 

1 

4.210938 

1.29663015 

3.248 

0.0016 

6 

1 

3.507813 

1.29663015 

2.705 

0.0081 

H 

1 

3.507813 

1.29663015 

2.705 

0.0081 

JK 

1 

-3.289063 

1.29663015 

-2 . 537 

0.0128 

FK 

1 

3.179688 

1.29663015 

2.452 

0.0160 

CE 

1 

-3.054688 

1.29663015 

-2.356 

0.0206 

BH 

1 

2.820313 

1.29663015 

2.175 

0.0321 

GK 

1 

2.820313 

1.29663015 

2.175 

0.0321 

HK 

1 

-2.742188 

1.29663015 

-2.115 

0.0371 

DG 

1 

-2.492188 

1.29663015 

-1 . 922 

0.0576 

BK 

1 

2.367188 

1.29663015 

1.826 

0.0711 

FG 

1 

-2.242188 

1.29663015 

-1.729 

0.0870 

BG 

1 

-2.023438 

1.29663015 

-1.561 

0.1220 

C 

1 

1.882813 

1.29663015 

1.452 

0.1498 

F 

1 

1.835938 

1.29663015 

1.416 

0.1601 

DH 

1 

1.789063 

1.29663015 

1.380 

0.1709 

A 

1 

-1.726563 

1.29663015 

-1.332 

0.1862 

AG 

1 

1.632813 

1.29663015 

1.259 

0.2110 

FH 

1 

1.507813 

1.29663015 

1.163 

0.2478 

BJ 

1 

-1.476563 

1.29663015 

-1.139 

0.2577 

J 

1 

1.398438 

1.29663015 

1.079 

0.2836 

AK 

1 

-1.351563 

1.29663015 

-1.042 

0.2999 

EK 

1 

-1.320313 

1.29663015 

-1.018 

0.3112 

AF 

1 

1.304688 

1.29663015 

1.006 

0.3169 

EG 

1 

-1.304688 

1.29663015 

-1.006 

0.3169 

A-29 


Attack  Casa  Max  Adj  Rsqr  Day  30 
Analysis  of  Variance 


Sum  of  Mean 

Souzca 

DF  Squares  Square 

F  Value 

Prob>F 

Modal 

27  39235. 

83594  1453.17911 

7 

.856 

0.0001 

Error 

100  18498. 

21875  184.98219 

C  total 

127  57734. 

05469 

Root 

MSE 

13.60082 

R- aqua re 

0.6796 

Dap  Maan 

42.91406 

Adj  R-sq 

0.5931 

C.V. 

31.69314 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob  >  I T | 

INTERCEP 

1 

42.914063 

1.20215363 

35.698 

0.0001 

B1 

1 

-16.476563 

3.18059953 

-5.180 

0.0001 

B2 

1 

-1.726563 

3.18059953 

-0.543 

0.5884 

B3 

1 

9.S85938 

3.18059953 

3.014 

0.0033 

B4 

1 

8.210938 

3.18059953 

2.582 

0.0113 

B5 

1 

8.335938 

3.18059953 

2.621 

0.0101 

B6 

1 

-2.226563 

3.18059953 

-0.700 

0.4855 

B7 

1 

-17.101563 

3.18059953 

-5.377 

0.0001 

B 

1 

7.554688 

1.20215363 

6.284 

0.0001 

FK 

1 

4.585938 

1.20215363 

3.815 

0.0002 

H 

1 

4.195313 

1.20215363 

3.490 

0.0007 

EJ 

1 

3.804688 

1.20215363 

3.165 

0.0021 

G 

1 

3.320313 

1.20215363 

2.762 

0.0068 

FG 

1 

-3.195313 

1.20215363 

-2.658 

0 . 0092 

BH 

1 

2.992188 

1.20215363 

2.489 

0.0145 

JK 

1 

-2.976563 

1.20215363 

-2.476 

0.0150 

BE 

1 

-2.382813 

1.20215363 

-1 . 982 

O'.  0502 

BG 

1 

-2.320313 

1 .20215363 

-1.930 

0.0564 

CE 

1 

-2.148438 

1.20215363 

-1.787 

0.0769 

HK 

1 

-2.054688 

1.20215363 

-1.709 

0.0905 

C 

1 

2.007813 

1.20215363 

1.670 

0.0980 

BK 

1 

1.929688 

1.20215363 

1.605 

0.1116 

GK 

1 

1.726563 

1.20215363 

1.436 

0.1541 

DH 

1 

1.664063 

1.20215363 

1.384 

0.1694 

EG 

1 

-1.554688 

1.20215363 

-1.293 

0.1989 

EK 

1 

-1.367188 

1.20215363 

-1.137 

0.2581 

AH 

1 

1.320313 

1.20215363 

1.098 

0.2747 

AK 

1 

-1.242188 

1.20215363 

-1.033 

0.3040 

A-30 


Araandg  R;  n»i1v  MfltomnHftlB  for  Prediction.  No- Attack  Case 


No-Attack  Case  Max  Adj  Rsqr  Day  1 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  3270. 

13281  130. 

80531 

6. 

423 

0.0001 

Error 

102  2077. 

17188  20. 

36443 

C 

Total 

127  5347. 

30469 

Root 

MSE 

4.51270 

R- aqua re 

0.6115 

Dep  Mean 

264.21094 

Adj  R-sq 

0.5163 

C.V. 

1.70799 

Parameter  Estimates 

Parameter 

Standard 

T 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  ! T I 

INTERCEP 

1 

264.210938 

0.39886979 

662.399 

0.0001 

B1 

1 

-1.210938 

1.05531027 

-1.147 

0.2539 

B2 

1 

-0.335938 

1.05531027 

-0.318 

0.7509 

B3 

1 

-0.710938 

1.05531027 

-0.674 

0.5020 

B4 

1 

-6.148438 

1.05531027 

-5.826 

0.0001 

BS 

1 

8.289063 

1.05531027 

7.855 

0.0001 

B6 

1 

-1.460937 

1.05531027 

-1.384 

0.1693 

B7 

1 

1.664063 

1.05531027 

1.577 

0.1179 

AC 

1 

1.492188 

0.39886979 

3.741 

0.0003 

FG 

1 

-1.304688 

0.39886979 

-3.271 

0.0015 

J 

1 

-1.210938 

0.39886979 

-3.036 

0.0030 

AG 

1 

-0.898438 

0.39886979 

-2.252 

0.0264 

CD 

1 

-0.835938 

0 . 39886979 

-2.096 

0.0386 

CF 

1 

-0.820313 

0.39886979 

-2.057 

0.0423 

A 

1 

-0.695313 

0.39886979 

-1.743 

0.0843 

CJ 

1 

-0.679688 

0.39886979 

-1.704 

0.0914 

BF 

1 

-0.632813 

0.39886979 

-1.587 

0.1157 

H 

1 

0.632813 

0.39886979 

1.587 

0.1157 

BD 

1 

0.601563 

0.39886979 

1.508 

0.1346 

FK 

1 

0.585938 

0.39886979 

1.469 

0.1449 

E 

1 

0.570313 

0.39886979 

1.430 

0.1558 

CK 

1 

-0.476563 

0.39886979 

-1.195 

0.2349 

F 

1 

-0.476563 

0.39886979 

-1.195 

0.2349 

DG 

1 

-0.445313 

0.39886979 

-1.116 

0.2669 

FJ 

1 

-0.429688 

0.39886979 

-1.077 

0.2839 

G 

1 

0.414063 

0.39886979 

1.038 

0.3017 

B-l 


No-Attack  Case  Max  Adj  Raqr  Day  2 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  28873. 

00781  1154.92031 

7 

.049 

0.0001 

Error 

102  16711. 

73438  163.84053 

C  Total 

127  45584. 

74219 

Root 

MSE 

12.80002 

R-square 

0.6334 

Dep  Mean 

212.86719 

Adj  R-sq 

0.5435 

C.V. 

6.01315 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

»  >  ITI 

INTERCEP 

1 

212.867188 

1.13137269 

188.149 

0.0001 

B1 

1 

-1.367188 

2.99333078 

-0.457 

0.6488 

B2 

1 

5.382813 

2.99333078 

1.798 

0.0751 

B3 

l 

-0.304688 

2.99333078 

-0.102 

0.9191 

B4 

1 

-2.867188 

2.99333078 

-0.958 

0.3404 

BS 

1 

-4.179688 

2.99333078 

-1.396 

0.1656 

B6 

1 

-1.617188 

2.99333078 

-0.540 

0.5902 

B7 

1 

4.007813 

2.99333078 

1.339 

0.1836 

E 

1 

12.195313 

1.13137269 

10.779 

0.0001 

DJ 

1 

-3.523438 

1.13137269 

-3.114 

0.0024 

GJ 

1 

-2.960938 

1.13137269 

-2.617 

0.0102 

EH 

1 

2.492188 

1.13137269 

2.203 

0.0299 

BJ 

1 

-2.273438 

1.13137269 

-2.009 

0.0471 

BD 

1 

-2.226563 

1.13137269 

-1.968 

0.0518 

K 

1 

2.164063 

1.13137269 

1.913 

0.0586 

A 

1 

2.039063 

1.13137269 

1.802 

0.0745 

FH 

1 

-1.773438 

1.13137269 

-1.568 

0.1201 

AE 

1 

-1.726563 

1.13137269 

-1.526 

0.1301 

EG 

1 

-1.585938 

1.13137269 

-1.402 

0.1640 

BF 

1 

-1.570313 

1.13137269 

-1.388 

0.1682 

DE 

1 

-1.398438 

1.13137269 

-1.236 

0.2193 

AC 

1 

-1.320313 

1.13137269 

-1.167 

0.2459 

AD 

1 

-1.304688 

1.13137269 

-1.153 

0.2515 

AK 

1 

1.273438 

1.13137269 

1.126 

0.2630 

AJ 

1 

-1.257813 

1.13137269 

-1.112 

0.2689 

CD 

1 

-1.117188 

1.13137269 

-0.987 

0.3258 

B-2 


No-Attack  Casa  Max  Ad j  Rsqr  Day  3 
Analysis  o f  Variance 


Source 

DF 

Sum  of 
Squares 

Mean 

Square 

F  Value 

Model 

28 

22869.09375 

816.75335 

6.936 

Error 

99 

11657.89844 

117.75655 

C  Total 

127 

34526.99219 

Root 

MSE 

10.85157 

R-square 

0.6624 

Dep  Mean 

C.V. 

210.00781  Adj  R-sq 

5.16722 

Parameter  Estimates 

Parameter  Standard 

0.5669 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

INTERCEP 

1 

210.007813 

0.95915225 

218.951 

B1 

1 

2.804688 

2.53767833 

1.105 

B2 

1 

6.554688 

2.53767833 

2.583 

B3 

1 

5.179688 

2.53767833 

2.041 

B4 

1 

3.242188 

2.53767833 

1.278 

BS 

1 

-5.445313 

2.53767833 

-2.146 

B6 

1 

-8.320312 

2.53767833 

-3.279 

B7 

1 

-4.132813 

2.53767833 

-1.629 

E 

1 

9.585938 

0.95915225 

9.994 

HK 

1 

3.304688 

0.95915225 

3.445 

AK 

1 

2.289063 

0.95915225 

2.387 

DH 

1 

-2.085938 

0.95915225 

-2.175 

DJ 

1 

1.914063 

0.95915225 

1.996 

D 

1 

1.914063 

0.95915225 

1.996 

A 

1 

-1.914063 

0.95915225 

-1.996 

BK 

1 

1.867188 

0.95915225 

1.947 

BF 

1 

-1.726563 

0.95915225 

-1.800 

EG 

1 

1.710938 

0.95915225 

1.784 

AB 

1 

-1.695313 

0.95915225 

-1.768 

BJ 

1 

-1.648438 

0.95915225 

-1.719 

H 

1 

1.601563 

0.95915225 

1.670 

BH 

1 

1.445313 

0.95915225 

1.507 

HJ 

1 

1.445313 

0.95915225 

1.507 

AH 

1 

1.335938 

0.95915225 

1.393 

DG 

1 

-1.242188 

0.95915225 

-1.295 

DF 

1 

1.148438 

0.95915225 

1.197 

DK 

1 

1.117188 

0.95915225 

1.165 

AE 

1 

1.070313 

0.95915225 

1.116 

CD 

1 

0.992188 

0.95915225 

1.034 

Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


4 


B-3 


Prob>F 

0.0001 


>  I T  I 

.0001 

.2717 

.0113 

.0439 

.2044 

.0343 

.0014 

.1066 

.0001 

.0008 

.0189 

.0320 

.0487 

.0487 

.0487 

.0544 

.0749 

.0775 

.0802 

.0888 

.0981 

.1350 

.1350 

.1668 

.1983 

.2340 

.2469 

.2672 

.3034 
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No-Attack  Case  Max  Adj  Raqr  Day  4 
Analysis  of  Variance 


Sura  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Prob>F 

Model 

29 

18283.22656 

629.76643 

4.376 

0.0001 

Error 

98 

14103.20313 

143.91024 

C  Total 

127 

32366.42969 

r,.v 


Variable  DF 


INTERCEP 


BH 


:  MSB 

11.99626 

R-aquare 

0.5643 

Mean 

205.22656 

Adj  R-aq 

0.4353 

5.84537 

Param 

eter  Estimates 

Parameter 

Standard 

T  for  HO: 

DF 

Estimate 

Error 

Parameters 

Prob  >  I T | 

1 

205.226563 

1.06032953 

193.550 

0.0001 

1 

0.148437 

2.80536825 

0.053 

0.9579 

1 

-3.351563 

2.80536825 

-1.195 

0.2351 

1 

0.085937 

2.80536825 

0.031 

0.9756 

1 

1.773438 

2.80536825 

0.632 

0.5288 

1 

-8.164063 

2.80536825 

-2.910 

0.0045 

1 

-6.226563 

2.80536825 

-2.220 

0.0288 

1 

7.960938 

2.80536825 

2.838 

0.0055 

1 

5.820313 

1.06032953 

5.489 

0.0001 

1 

4.273438 

1.06032953 

4.030 

0.0001 

1 

-4.179688 

1.06032953 

-3.942 

0.0002 

1 

2.507813 

1.06032953 

2.365 

0.0200 

1 

-2.085938 

1 . 06032953 

-1.967 

0.0520 

1 

-1.992188 

1.06032953 

-1.879 

0.0632 

1 

1.742188 

1.06032953 

1.643 

0.1036 

1 

1.710938 

1.06032953 

1.614 

0.1098 

1 

1.554688 

1.06032953 

1.466 

0.1458 

1 

-1.507813 

1.06032953 

-1.422 

0.1582 

1 

-1.429688 

1.06032953 

-1.348 

0.1807 

1 

1.429688 

1.06032953 

1.348 

0.1807 

1 

1.257813 

1.06032953 

1.186 

0.2384 

1 

-1.210938 

1.06032953 

-1.142 

0.2562 

1 

1.210938 

1.06032953 

1.142 

0.2562 

1 

-1.195313 

1.06032953 

-1.127 

0.2624 

1 

-1.195313 

1.06032953 

-1.127 

0.2624 

1 

1.101563 

1.06032953 

1.039 

0.3014 

1 

-1.085938 

1.06032953 

-1,024 

0.3083 

1 

-1.085938 

1.06032953 

-1.024 

0.3083 

1 

-1.085938 

1.06032953 

-1.024 

0.3083 

1 

1.070313 

1.06032953 

1.009 

0.3153 

B-4 


No-Attack  Casa  Max  Adj  Raqr  Day  5 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Model 

28 

23635.90625 

844.13951 

5.315 

Error 

"99 

15724.83594 

158.83673 

C  Total 

127 

39360.74219 

Root  MSE 

12 . 60304  R-square 

0.6005 

Dep  Mean 

198.13281  Adj 

R-sq 

0.4875 

C.V. 

6.36091 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob 

INTERCEP 

1 

198.132813 

1.11396226 

177.863 

0 

B1 

1 

-3.445313 

2.94726712 

-1.169 

0 

B2 

1 

-3.382813 

2.94726712 

-1.148 

0 

B3 

1 

4.054688 

2.94726712 

1.376 

0 

B4 

1 

9.429688 

2.94726712 

3.199 

0 

B5 

1 

-1.695313 

2.94726712 

-0.575 

0 

B6 

1 

-0.445313 

2.94726712 

-0.151 

0 

B7 

1 

2.429688 

2.94726712 

0.824 

0 

E 

1 

7.492188 

1.11396226 

6.726 

0 

B 

1 

6.242188 

1.11396226 

5.604 

0 

BH 

1 

-3.023438 

1.11396226 

-2.714 

0 

CG 

1 

-2.585938 

1.11396226 

-2.321 

0 

H 

1 

2.554688 

1.11396226 

2.293 

0 

BJ 

1 

2.242188 

1.11396226 

2.013 

0 

EJ 

1 

2.117188 

1.11396226 

1.901 

0 

EF 

1 

2.023438 

1.11396226 

1.816 

0 

AK 

1 

1.757813 

1.11396226 

1.578 

0 

CK 

1 

1.742188 

1.11396226 

1.564 

0 

C 

1 

1.742188 

1.11396226 

1.564 

0 

BK 

1 

-1.726563 

1.11396226 

-1.550 

0 

J 

1 

1.726563 

1.11396226 

1.550 

0 

AC 

1 

1.648438 

1.11396226 

1.480 

0 

FH 

1 

-1.570313 

1.11396226 

-1.410 

0 

CE 

1 

-1.523438 

1.11396226 

-1.368 

0 

D 

1 

-1.445313 

1.11396226 

-1.297 

0 

GH 

1 

-1.242188 

1.11396226 

-1.115 

0 

HJ 

1 

1.210938 

1.11396226 

1.087 

0 

AJ 

1 

1.164063 

1.11396226 

1.045 

0 

AD 

1 

-1.132813 

1.11396226 

-1.017 

0 

Prob>F 

0.0001 


>  I T I 

.0001 

.2452 

.2538 

.1720 

.0019 

.5665 

.8802 

.4117 

.0001 

.0001 

.0078 

.0223 

.0239 

.0469 

.0603 

.0723 

.1178 

.1210 

.1210 

.1243 

.1243 

.1421 

.1618 

.1745 

.1975 

.2675 

.2797 

.2986 

.3117 


Mo-Attack  Case  Max  Adj  Rsqr  Day  6 
Analysis  of  Variance 


Sum  of  Moan 

Sourcs 

DF  Squares  Square 

F  Value 

Prob>F 

Modal 

36  23821. 

65625  661.71267 

8 

.361 

0.0001 

Error 

91  7202. 

34375  79.14663 

C  Total 

127  31024. 

00000 

Root 

MSE 

8.89644 

R-square 

0.7678 

Dap  Maan 

188.50000 

Adj  R-sq 

0.6760 

C.V. 

4.71960 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

>  IT | 

INTERCEP 

1 

188.500000 

0.78634158 

239.718 

0.0001 

B1 

1 

-1.125000 

2.08046427 

-0.541 

0.5900 

B2 

1 

0.437500 

2.08046427 

0.210 

0.8339 

B3 

1 

5.500000 

2.08046427 

2.644 

0.0097 

B4 

1 

0.062500 

2.08046427 

0.030 

0.9761 

BS 

1 

0.500000 

2.08046427 

0.240 

0.8106 

B6 

1 

0.937500 

2.08046427 

0.451 

0.6533 

B7 

1 

-0.375000 

2.08046427 

-0.180 

0.8574 

E 

1 

7.640625 

0.78634158 

9.717 

0.0001 

B 

1 

5.843750 

0.78634158 

7.432 

0.0001 

BJ 

1 

3.359375 

0.78634158 

4.272 

0.0001 

CJ 

1 

-3.234375 

0.78634158 

-4.113 

0.0001 

H 

1 

2.796875 

0.78634158 

3.557 

0.0006 

FK 

1 

2.718750 

0.78634158 

3.457 

0.0008 

J 

1 

2.234375 

0.78634158 

2.841 

0.0055 

AK 

1 

2.234375 

0.78634158 

2.841 

0.0055 

A 

1 

1.828125 

0.78634158 

2.325 

0.0223 

C 

1 

1.781250 

0.78634158 

2.265 

0.0259 

AD 

1 

-1.750000 

0.78634158 

-2.225 

0.0285 

HK 

1 

1.671875 

0.78634158 

2.126 

0.0362 

EF 

1 

1.609375 

0.78634158 

2.047 

0.0436 

BE 

1 

1.546875 

0.78634158 

1.967 

0.0522 

AB 

1 

-1.546875 

0.78634158 

-1.967 

0.0522 

CF 

1 

-1.375000 

0.78634158 

-1.749 

0.0837 

BC 

1 

1.343750 

0.78634158 

1.709 

0.0909 

BG 

1 

-1.312500 

0.78634158 

-1.669 

0.0985 

BD 

1 

1.296875 

0.78634158 

1.649 

0.1025 

CE 

1 

-1.265625 

0.78634158 

-1.610 

0.1110 

GK 

1 

1.250000 

0.78634158 

1.590 

0.1154 

CH 

1 

-1.171875 

0.78634158 

-1.490 

0.1396 

EG 

1 

1.171875 

0.78634158 

1.490 

0.1396 

EK 

1 

1.109375 

0.78634158 

1.411 

0.1617 

AE 

1 

1.062500 

0.78634158 

1.351 

0.1800 

AH 

1 

0.906250 

0.78634158 

1.152 

0.2521 

AG 

1 

0.859375 

0.78634158 

1.093 

0.2773 

K 

1 

-0.843750 

0.78634158 

-1.073 

0.2861 

CK 

1 

0.843750 

0.78634158 

1.073 

0.2861 

B-6 


No-Attack  Case  Max  Ad j  Raqr  Day  7 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Prob>F 

Model 

29 

24463.53906 

843.57031 

4.838 

0.0001 

Error 

98 

17086.39063 

174.35092 

C  Total 

127 

41549.92969 

Root  MSB 

13.20420  R-square 

0.5888 

Dep  Mean 

185.52344  Adj 

R-aq 

0.4671 

C.V. 

7.11727 

Parameter  Estimates 


Variable 

DF 

Parameter 

Estimate 

Standard 

Error 

T  for  HO: 
Parameters 

Prob  >  I T I 

INTERCEP 

1 

185.523438 

1.16709751 

158.961 

0.0001 

B1 

1 

0.226562 

3.08784977 

0.073 

0.9417 

B2 

1 

8.976563 

3.08784977 

2.907 

0.0045 

B3 

1 

-4.960938 

3.08784977 

-1.607 

0.1114 

B4 

1 

5.601563 

3.08784977 

1.814 

0.0727 

B5 

1 

-1.773438 

3.08784977 

-0.574 

0.5671 

B6 

1 

-10.835938 

3.08784977 

-3.509 

0.0007 

B7 

1 

2.914063 

3.08784977 

0.944 

0.3476 

B 

1 

7.882813 

1.16709751 

6.754 

0.0001 

E 

1 

5.273438 

1.16709751 

4.518 

0.0001 

EJ 

1 

3.132813 

1.16709751 

2.684 

0.0085 

CE 

1 

-2.835938 

1.16709751 

-2.430 

0.0169 

AG 

1 

2.148438 

1.16709751 

1.841 

0.0687 

H 

1 

1.960938 

1.16709751 

1.680 

0.0961 

DK 

1 

-1.945313 

1.16709751 

-1.667 

0.0987 

EH 

1 

-1.945313 

1.16709751 

-1.667 

0.0987 

AF 

1 

-1.898438 

1.16709751 

-1.627 

0.1070 

CH 

1 

-1.898438 

1.16709751 

-1.627 

0.1070 

BD 

1 

1.882813 

1.16709751 

1.613 

0.1099 

BG 

1 

1.804688 

1.16709751 

1.546 

0.1253 

EK 

1 

1.773438 

1.16709751 

1.520 

0.1318 

HJ 

1 

1.601563 

1.16709751 

1.372 

0.1731 

G 

1 

-1.554688 

1.16709751 

-1.332 

0.1859 

BJ 

1 

1.398438 

1.16709751 

1.198 

0.2337 

AE 

1 

-1.398438 

1.16709751 

-1.198 

0.2337 

AK 

1 

1.382813 

1.16709751 

1.185 

0.2389 

DH 

1 

-1.351563 

1.16709751 

-1.158 

0.2497 

AJ 

1 

-1.226563 

1.16709751 

-1.051 

0.2959 

EF 

1 

1.179688 

1.16709751 

1.011 

0.3146 

EG 

1 

-1.179688 

1.16709751 

-1.011 

0.3146 

B-7 


No-Attack  Casa  Max  Ad j  Raqr  Day  8 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

32  60094. 

15625  1877.94238 

4. 

660 

0.0001 

Error 

95  38282. 

71875  402.97599 

C  Total 

127  98376. 

87500 

Root 

MSE 

20.07426 

R- square 

0.6109 

Dep  Mean 

171.40625 

Adj  R-sq 

0.4798 

C.V. 

11.71151 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob 

•  >  1 T  | 

INTERCEP 

1 

171.406250 

1.77433083 

96.603 

0.0001 

B1 

1 

-1.281250 

4.69443812 

-0.273 

0.7855 

B2 

1 

-0.968750 

4.  9443812 

-0.206 

0.8370 

B3 

1 

-2.968750 

4.o9443812 

-0.632 

0.5286 

B4 

1 

12.843750 

4.69443812 

2.736 

0.0074 

B5 

1 

-3.906250 

4.69443812 

-0.832 

0.4074 

B6 

1 

-6.406250 

4.69443812 

-1.365 

0.1756 

B7 

1 

-0.468750 

4.69443812 

-0.100 

0.9207 

J 

1 

9.687500 

1.77433083 

5.460 

0.0001 

BJ 

1 

9.406250 

1.77433083 

5.301 

0.0001 

EJ 

1 

8.187500 

1.77433083 

4.614 

0.0001 

GH 

1 

-4.703125 

1.77433083 

-2 . 651 

0.0094 

EU 

1 

-4.656250 

1.77433083 

-2 . 624 

0.0101 

B 

1 

4.593750 

1.77433083 

2.589 

0.0111 

BG 

1 

4.015625 

1.77433083 

2.263 

0.0259 

F 

1 

3.281250 

1.77433083 

1.849 

0.0675 

AG 

1 

3.250000 

1.77433083 

1.832 

0.0701 

BC 

1 

-2.796875 

1.77433083 

-1.576 

0.1183 

DH 

1 

2.765625 

1.77433083 

1.559 

0.1224 

BE 

1 

-2.656250 

1.77433083 

-1.497 

0.1377 

CE 

1 

-2.609375 

1.77433083 

-1.471 

0.1447 

DF 

1 

2.515625 

1.77433083 

1.418 

0.1595 

AK 

1 

2.500000 

1.77433083 

1.409 

0.1621 

HK 

1 

-2.390625 

1.77433083 

-1.347 

0.1811 

AJ 

1 

2.359375 

1.77433083 

1.330 

0.1868 

BF 

1 

2.187500 

1.77433083 

1.233 

0.2207 

BK 

1 

2.171875 

1.77433083 

1.224 

0.2240 

JK 

1 

-2.140625 

1.77433083 

-1.206 

0.2306 

D 

1 

-2.140625 

1.77433083 

-1.206 

0.2306 

G 

1 

2.109375 

1.77433083 

1.189 

0.2375 

FJ 

1 

-2.031250 

1.77433083 

-1.145 

0.2552 

E 

1 

-1.906250 

1.77433083 

-1.074 

0.2854 

DE 

1 

-1.828125 

1.77433083 

-1.030 

0.3055 

B-8 


No-Attack  Case  Max  Adj  Rsqr  Day  9 
Analysis  of  Variance 


Sum  of  Mean 

Source  DF  Squares  Square  F  Value  Prob>F 


Model 

33  186442. 

28125  5649.76610 

11 

.082  0.0001 

Error 

94  47921. 

21875  509.80020 

C  Total 

127  234363. 

50000 

Root 

MSE 

22.57876 

R-square 

0.7955 

Dep  Mean 

151.06250 

Adj  R-sq 

0.7237 

C.V. 

14.94663 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  I T I 

INTERCEP 

1 

151.062500 

1.99569889 

75.694 

0.0001 

B1 

1 

0.312500 

5.28012295 

0.059 

0.9529 

B2 

1 

0.062500 

5.28012295 

0.012 

0.9906 

B3 

1 

0.562500 

5.28012295 

0.107 

0.9154 

B4 

1 

6.375000 

5.28012295 

1  207 

0.2303 

B5 

1 

-10.312500 

5.28012295 

-1.953 

0.0538 

B6 

1 

1.753000 

5.28012295 

0.331 

0.7411 

B7 

1 

5.500000 

5.28012295 

1.042 

0.3002 

J 

1 

27.609375 

1.99569889 

13.834 

0.0001 

EJ 

1 

14.656250 

1.99569889 

7.344 

0.0001 

BJ 

1 

12.531250 

1.99569889 

6.279 

0.0001 

E 

1 

-8.296875 

1.99569889 

-4.157 

0.0001 

EG 

1 

5.671875 

1.99569889 

2.842 

0.0055 

HK 

1 

-4.171875 

1.99569889 

-2.090 

0.0393 

DJ 

1 

3.812500 

1.99569889 

1.910 

0.0591 

AE 

1 

-3.578125 

1.99569889 

-1.793 

0.0762 

FG 

1 

-3.578125 

1.99569889 

-1.793 

0.0762 

AC 

1 

3.562500 

1.99569889 

1.785 

0.0775 

DE 

1 

-3.531250 

1.99569889 

-1.769 

0.0801 

AK 

1 

3.500000 

1.99569889 

1.754 

0.0827 

GH 

1 

-3.359375 

1.99569889 

-1.683 

0.0956 

GK 

1 

-3.281250 

1.99569889 

-1.644 

0.1035 

CF 

1 

3.171875 

1.99569889 

1.589 

0.1153 

R 

1 

2.984375 

1.99569889 

1.495 

0.1382 

AH 

1 

2.921875  . 

1.99569889 

1.464 

0.1465 

AB 

1 

2.890625 

1.99569889 

1.448 

0.1508 

EH 

1 

-2.750000 

1.99569889 

-1.378 

0.1715 

BG 

1 

-2.640625 

1.99569889 

-1.323 

0.1890 

AF 

1 

2.453125 

1.99569889 

1.229 

0.2221 

AD 

1 

2.359375 

1.99569889 

1.182 

0.2401 

FK 

1 

-2.203125 

1.99569889 

-1.104 

0.2724 

D 

1 

-2.109375 

1.99569889 

-1.057 

0.2932 

FH 

1 

2.062500 

1.99569889 

1.033 

0.3040 

BK 

1 

2.015625 

1.99569889 

1.010 

0.3151 

A 


B-9 


No-Attack  Casa  Max  Adj  Rsqr  Day  10 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

29  47393. 

41406  1634.25566 

13 

.715 

0.0001 

Error 

98  11677. 

39062  1a9. 15705 

C  Total 

127  59070. 

80469 

Root 

MSE 

10.91591 

R- square 

0.8023 

Dep  Mean 

168.03906 

Adj  R-sq 

0.7438 

C.V. 

6.49605 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob 

»  >  1  T  I 

INTERCEP 

1 

168.039063 

0.96483907 

174.163 

0.0001 

B1 

1 

-1.039063 

2.55272423 

-0.407 

0.6849 

B2 

1 

-0.601563 

2.55272423 

-0.236 

0.8142 

B3 

1 

-3.726563 

2.55272423 

-1.460 

0.1475 

B4 

1 

4.398438 

2.55272423 

1.723 

0.0880 

B5 

1 

1.148438 

2.55272423 

0.450 

0.6538 

B6 

1 

-1.039063 

2.55272423 

-0.407 

0.6849 

B7 

1 

6.398438 

2.55272423 

2.507 

0.0138 

B 

1 

15.257813 

0.96483907 

15.814 

0.0001 

J 

1 

5.179688 

0.96483907 

5.368 

0.0001 

EJ 

1 

4.835938 

0.96483907 

5.012 

0.0001 

H 

1 

3.367188 

0.96483907 

3.490 

0.0007 

BJ 

1 

3.117188 

'<.96483907 

3.231 

0.0017 

F 

1 

3.039063 

0.96483907 

3.150 

0.0022 

AG 

1 

2.539063 

0.96483907 

2.632 

0.0099 

EG 

1 

2.179688 

0.96483907 

2.259 

0.0261 

EF 

1 

-2.117188 

0.96483907 

-2.194 

0.0306 

GH 

1 

-1.992188 

0.96483907 

-2.065 

0.0416 

E 

1 

1.820313 

0.96483907 

1.887 

0.0622 

AC 

1 

1.757813 

0.96483907 

1.822 

0.0715 

FG 

1 

-1.726563 

0.96483907 

-1.789 

0.0766 

AB 

1 

-1.601563 

0.96483907 

-1.660 

0.1001 

FK 

1 

1.476563 

0.96483907 

1.530 

0.1291 

EK 

1 

-1.304688 

0.96483907 

-1.352 

0.1794 

DE 

1 

■  -1.257813 

0.96483907 

-1.304 

0.1954 

GK 

1 

1.210938 

0.96483907 

1.255 

0.2124 

AH 

1 

-1.179688 

0.96483907 

-1.223 

0.2244 

CG 

1 

-1.117188 

0.96483907 

-1.158 

0.2497 

HJ 

1 

1.101563 

0.96483907 

1.142 

0.2564 

BE 

1 

-0.992188 

0.96483907 

-1.028 

0.3063 

B-10 


No- Attack  Casa  Max  Adj  Raqr  Day  11 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

25  65487. 

21875  2619.48875 

14 

.037 

0.0001 

Error 

102  19034. 

50000  186.61275 

C 

Total 

127  84521. 

71875 

Root 

MSE 

13.66063 

R-square 

0.7748 

Dep  Mean 

173.54688 

Adj  R-sq 

0.7196 

C.V. 

7.87143 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob  >  I T | 

INTERCEP 

1 

173.546875 

1.20744030 

143.731 

0.0001 

B1 

1 

4.203125 

3.19458675 

1.316 

0.1912 

B2 

1 

1.328125 

3.19458675 

0.416 

0.6785 

B3 

1 

-0.734375 

3.19458675 

-0.230 

0.8186 

B4 

1 

-2.484375 

3.19458675 

-0.778 

0.4386 

B5 

1 

2.203125 

3.19458675 

0.690 

0.4920 

B6 

1 

0.015625 

3.19458675 

0.005 

0.9961 

B7 

1 

1.015625 

3.19458675 

0.318 

0.7512 

B 

1 

19.781250 

1.20744030 

16.383 

0.0001 

H 

1 

5.296875 

1.20744030 

4.387 

0.0001 

J 

1 

-5.031250 

1.20744030 

-4.167 

0.0001 

E 

1 

3.296875 

1.20744030 

2.730 

0.0075 

HJ 

1 

-3.062500 

1.20744030 

-2.536 

0.0127 

K 

1 

-2.515625 

1.20744030 

-2.083 

0.0397 

AB 

1 

-2.343750 

1.20744030 

-1.941 

0.0550 

CK 

1 

1.875000 

1.20744030 

1.553 

0.1236 

AC 

1 

-1.843750 

1.20744030 

-1.527 

0.1299 

CJ 

1 

-1.734375 

1.20744030 

-1.436 

0.1539 

BH 

1 

-1.718750 

1.20744030 

-1.423 

0.1577 

GJ 

1 

1.718750 

1.20744030 

1.423 

0.1577 

BD 

1 

1.593750 

1.20744030 

1.320 

0.1898 

EK 

1 

1.515625 

1.20744030 

1.255 

0.2123 

BC 

1 

1.484375 

1.20744030 

1.229 

0.2218 

CD 

1 

1.218750 

1.20744030 

1.009 

0.3152 

FH 

1 

1.171875 

1.20744030 

0.971 

0.3341 

DE 

1 

-1.140625 

1.20744030 

-0.945 

0.3471 

No-Attack  Case  Max  Adj  Raqr  Day  12 
Analysis  of  Variancs 


Sum  of  Mean 

Sourcs 

DF  Squares  Square 

F  Value 

Prob>F 

Modal 

29  67332. 

09375  2321.79634 

12 

.305 

0.0001 

Error 

98  18491. 

78125  188.69165 

C 

Total 

127  85823. 

87500 

Root 

MSB 

13.73651 

R-square 

0.7845 

Dap  Maan 

166.78125 

Adj  R-sq 

0.7208 

C.V. 

8.23624 

Parameter  Estimates 

Paramater 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  I T I 

INTERCEP 

1 

166.781250 

1.21414722 

137.365 

0.0001 

B1 

1 

0.468750 

3.21233161 

0.146 

0.8843 

B2 

1 

4.593750 

3.21233161 

1.430 

0.1559 

S3 

1 

2.093750 

3.21233161 

0.652 

0.5161 

B4 

1 

-3.593750 

3.21233161 

-1.119 

0.2660 

BS 

1 

-1.906250 

3.21233161 

-0.593 

0.5543 

B6 

1 

-0.093750 

3.21233161 

-0.029 

0.9768 

B7 

1 

4.406250 

3.21233161 

1.372 

0.1733 

B 

1 

20.265625 

1.21414722 

16.691 

0.0001 

H 

1 

4.656250 

1.21414722 

3.835 

0.0002 

GJ 

1 

2.796875 

1.21414722 

2.304 

0.0234 

E 

1 

2.609375 

1.21414722 

2.149 

0.0341 

GH 

1 

2.531250 

1.21414722 

2.085 

0.0397 

EG 

1 

2.515625 

1.21414722 

2.072 

0.0409 

AE 

1 

-2.468750 

1.21414722 

-2.033 

0.0447 

DG 

1 

-2.437500 

1.21414722 

-2.008 

0.0474 

AF 

1 

2.296875 

1.21414722 

1.892 

0.0615 

AC 

1 

-2.281250 

1.21414722 

-1.879 

0.0632 

DH 

1 

2.156250 

1.21414722 

1.776 

0.0788 

F 

1 

1.843750 

1.21414722 

1.519 

0.1321 

FG 

1 

-1.843750 

1.21414722 

-1.519 

0.1321 

FK 

1 

1.703125 

1.21414722 

1.403 

0.1639 

HK 

1 

1.671875 

1.21414722 

1.377 

0.1717 

AD 

1 

-1.546875 

1.21414722 

-1.274 

0.2057 

CJ 

1 

-1.531250 

1.21414722 

-1.261 

0.2102 

G 

1 

-1.531250 

1.21414722 

-1.261 

0.2102 

A 

1 

1.515625 

1.21414722 

1.248 

0.2149 

BJ 

1 

1.312500 

1.21414722 

1.081 

0.2823 

HJ 

1 

-1.296875 

1.21414722 

-1.068 

0.2881 

DJ 

1 

1.296875 

1.21414722 

1.068 

0.2881 

B-12 
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No-Attack  Casa  Max  Adj  Raqr  Day  13 


Souzce 


Analysis  of 

Variance 

Sum  of 

Mean 

Squares 

Square 

F  Value 

Model 

30  64144. 

60938  2138.15365 

15 

Error 

97  13718. 

44531  141.42727 

C  Total 

127  77863. 

05469 

Root 

MSE 

11.89232 

R- aqua re 

0.8238 

Dap  Mean 

160.41406 

Adj  R-sq 

0.7693 

C.V. 

7.41351 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

INTERCEP 

1 

160.414063 

1.05114250 

152.609 

B1 

1 

1.148437 

2.78106165 

0.413 

B2 

1 

1.460937 

2.78106165 

0.525 

B3 

1 

-1.039063 

2.78106165 

-0.374 

B4 

1 

-0.539063 

2.78106165 

-0.194 

B5 

1 

1.898438 

2.78106165 

0.683 

B6 

1 

-7.351563 

2.78106165 

-2.643 

B7 

1 

7.710938 

2.78106165 

2.773 

B 

1 

19.679688 

1.05114250 

18.722 

H 

1 

4.539063 

1.05114250 

4.318 

E 

1 

3.898438 

1.05114250 

3.709 

DK 

1 

2.585938 

1.05114250 

2.460 

CE 

1 

2.367188 

1.05114250 

2.252 

FH 

1 

2.304688 

1.05114250 

2.193 

AJ 

1 

2.210938 

1.05114250 

2.103 

CD 

1 

2.164063 

1.05114250 

2.059 

AC 

1 

-2.101563 

1.05114250 

-1.999 

J 

1 

-1.929688 

1.05114250 

-1.836 

C 

1 

1.789063 

1.05114250 

1.702 

CH 

1 

1.757813 

1.05114250 

1.672 

GH 

1 

1.710938 

1.05114250 

1.628 

DE 

1 

-1.632813 

1.05114250 

-1.553 

AE 

1 

1.476563 

1.05114250 

1.405 

EJ 

1 

-1.382813 

1.05114250 

-1.316 

BK 

1 

-1.335938 

1.05114250 

-1.271 

BE 

1 

1.257813 

1.05114250 

1.197 

HJ 

1 

-1.242188 

1.05114250 

-1.182 

GK 

1 

-1.179688 

1.05114250 

-1.122 

BH 

1 

-1.164063 

1.05114250 

-1.107 

BD 

1 

1.148438 

1.05114250 

1.093 

DH 

1 

-1.085938 

1.05114250 

-1.033 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


Prob>F 

0.0001 


>  ITI 

.0001 

.6806 

.6006 

.7095 

.8467 

.4965 

.0096 

.0067 

.0001 

.0001 

.0003 

.0157 

.0266 

.0307 

.0380 

.0422 

.0484 

.0695 

.0920 

.0977 

.1068 

.1236 

.1633 

.1914 

.2068 

.2344 

.2402 

.2645 

1.2708 

1.2773 

1.3043 
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No-Attack  Caae  Max  Ad j  Raqr  Day  14 
Analyaia  of  Variance 


Sum  of 

Mean 

Source 

DF  Squarea  Square 

F  Value 

Prob>F 

Model 

25  55373. 

63281  2214. 

94531 

10. 

956 

0.0001 

Error 

102  20621. 

23437  202. 

16896 

C 

Total 

127  75994. 

86719 

Root 

MSE 

14.21861 

R-aquare 

0.7286 

Dep  Mean 

153.82031 

Adj  R-aq 

0.6621 

C.V. 

9.24365 

Parameter  Eatlmatea 

Paramo  tor 

Standard 

T 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T I 

INTERCEP 

1 

153.820313 

1.25675974 

122.394 

0.0001 

B1 

1 

7.617188 

3.32507372 

2.291 

0.0240 

B2 

1 

-1.695313 

3.32507372 

-0.510 

0.6113 

B3 

1 

-2.695313 

3.32507372 

-0.811 

0.4195 

B4 

1 

-0.195313 

3.32507372 

-0.059 

0.9533 

B5 

1 

0.179688 

3.32507372 

0.054 

0.9570 

B6 

1 

-3.445313 

3.32507372 

-1.036 

0.3026 

B7 

1 

-3.382813 

3.32507372 

-1.017 

0.3114 

B 

1 

17.945313 

1.25675974 

14.279 

0.0001 

H 

1 

5.867188 

1.25675974 

4.669 

0.0001 

BE 

1 

3.820313 

1.25675974 

3.040 

0.0030 

E 

1 

2.914063 

1.25675974 

2.319 

0.0224 

BD 

1 

2.507813 

1.25675974 

1.995 

0.0487 

DH 

1 

-2.289063 

1.25675974 

-1.821 

0.0715 

CJ 

1 

-1.929688 

1.25675974 

-1.535 

0.1278 

EJ 

1 

-1.773438 

1.25675974 

-1.411 

0.1613 

DE 

1 

1.664063 

1.25675974 

1.324 

0.1884 

AG 

1 

1.648438 

1.25675974 

1.312 

0.1926 

AF 

1 

-1.554688 

1.25675974 

-1.237 

0.2189 

GH 

1 

1.507813 

1.25675974 

1.200 

0.2330 

F 

1 

1.445313 

1.25675974 

1.150 

0.2528 

JK 

1 

1.429688 

1.25675974 

1.138 

0.2580 

CH 

1 

-1.414063 

1.25675974 

-1.125 

0.2632 

CD 

1 

1.382813 

1.25675974 

1.100 

0.2738 

EK 

1 

1.304688 

1.25675974 

1.038 

0.3017 

HJ 

1 

-1.289063 

1.25675974 

-1.026 

0.3075 
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No-Attack  Case  Max  Adj  Raqr  Day  15 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

27  55465. 

58594  2054.28096 

12 

.834 

0.0001 

Error 

100  16006. 

28125  160.06281 

C  Total 

127  71471. 

86719 

Root 

MSE 

12.65159 

R-square 

0.7760 

Dep  Mean 

144.32031 

Adj  R-sq 

0.7156 

C.V. 

8.76633 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameters 

Prob  >  I T I 

INTERCEP 

1 

144.320313 

1.11825343 

129.059 

0.0001 

B1 

1 

-5.445313 

2.95862047 

-1.840 

0.0687 

B2 

1 

-0.132813 

2.95862047 

-0.045 

0.9643 

B3 

1 

-4.570313 

2.95862047 

-1.545 

0.1256 

B4 

1 

2.054688 

2.95862047 

0.694 

0.4890 

B5 

1 

-0.195313 

2.95862047 

-0.066 

0.9475 

B6 

1 

0.804688 

2.95862047 

0.272 

0.7862 

B7 

1 

0.304688 

2.95862047 

0.103 

0.9182 

B 

1 

18.132813 

1.11825343 

16.215 

0.0001 

H 

1 

5.085938 

1.11825343 

4.548 

0.0001 

AE 

1 

2.945313 

1.11825343 

2.634 

0.0098 

A 

1 

2.757813 

1.11825343 

2.466 

0.0154 

AB 

1 

2.726563 

1.11825343 

2.438 

0.0165 

CD 

1 

-2.304688 

1.11825343 

-2.061 

0.0419 

K 

1 

-2.257813 

1.11825343 

-2.019 

0.0462 

BJ 

1 

2.132813 

1.11825343 

1.907 

0.0594 

EH 

1 

-2.101563 

1.11825343 

-1.879 

0.0631 

BF 

1 

1.945313 

1.11825343 

1.740 

0.0850 

GK 

1 

-1.617188 

1.11825343 

-1.446 

0.1513 

FK 

1 

-1.507813 

1.11825343 

-1.348 

0.1806 

BH 

1 

1.460938 

1.11825343 

1.306 

0.1944 

BD 

1 

1.445313 

1.11825343 

1.292 

0.1992 

EK 

1 

-1.382813 

1.11825343 

-1.237 

0.2191 

DE 

1 

1.257813 

1.11825343 

1.125 

0.2634 

BC 

1 

1.226563 

1.11825343 

1.097 

0.2753 

DF 

1 

-1.210938 

1.11825343 

-1.083 

0.2815 

J 

1 

1.195313 

1.11825343 

1.069 

0.2877 

CK 

1 

1.179688 

1.11825343 

1.055 

0.2940 

B-15 


No-Attack  Case  Max  Ad j  Rsqr  Day  16 


Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

30  73586. 

54687  2452.88490 

14 

.521 

0.0001 

Error 

97  16384. 

75781  168.91503 

C  Total 

127  89971. 

30469 

Root 

MSE 

12.99673 

R-square 

0.8179 

Dep  Mean 

140.21094 

Adj  R-aq 

0.7616 

C.V. 

9.26941 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  1 T | 

INTERCEP 

1 

140.210938 

1.14875962 

122.054 

0.0001 

B1 

1 

4.601563 

3.03933227 

1.514 

0.1333 

B2 

1 

-5.523438 

3.03933227 

-1.817 

0.0723 

B3 

1 

-2.585938 

3.03933227 

-0.851 

0.3970 

B4 

1 

-0.960938 

3.03933227 

-0.316 

0.7526 

B5 

1 

2.539063 

3.03933227 

0.835 

0.4055 

B6 

1 

-0.085937 

3.03933227 

-0.028 

0.9775 

B7 

1 

0.726563 

3.03933227 

0.239 

0.8116 

B 

1 

21.523438 

1.14875962 

18.736 

0.0001 

H 

1 

3.804688 

1.14875962 

3.312 

0.0013 

BC 

1 

3.445313 

1.14875962 

2.999 

0.0034 

AJ 

1 

3.085938 

1.14875962 

2.686 

0.0085 

DK 

1 

2.648438 

1.14875962 

2.305 

0.0233 

CE 

1 

2.539063  . 

1.14875962 

2.210 

0.0294 

AB 

1 

-2.507813 

1.14875962 

-2.183 

0.0314 

BH 

1 

2.492188 

1.14875962 

2.169 

0.0325 

BE 

1 

2.148438 

1.14875962 

1.870 

0.0645 

6 

1 

2.117188 

1.14875962 

1.843 

0.0684 

EK 

1 

2.085938 

1.14875962 

1.816 

0.0725 

GK 

1 

-1.757813 

1.14875962 

-1.530 

0.1292 

E 

1 

1.679688 

1.14875962 

1.462 

0.1469 

FG 

1 

-1.679688 

1.14875962 

-1.462 

0.1469 

FH 

1 

1.632813 

1.14875962 

1.421 

0.1584 

HJ 

1 

-1.570313 

1.14875962 

-1.367 

0.1748 

FJ 

1 

-1.554688 

1.14875962 

-1.353 

0.1791 

DJ 

1 

1.460938 

1.14875962 

1.272 

0.2065 

CF 

1 

-1.445313 

1.14875962 

-1.258 

0.2114 

EJ 

1 

-1.445313 

1.14875962 

-1.258 

0.2114 

BD 

1 

1.335938 

1.14875962 

1.163 

0.2477 

CD 

1 

1.320313 

1.14875962 

1.149 

0.2532 

CG 

1 

1.320313 

1.14875962 

1.149 

0.2532 

B-16 


Mo-Attack  Casa  Max  Adj  Rsqr  Day  17 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

40  74098. 

93750  1852.47344 

13 

.442 

0.0001 

Error 

87  11989. 

61719  137.81169 

C  Total 

127  86088. 

55469 

Root 

MSE 

11.73932 

R- square 

0.8607 

Dep  Mean 

135.33594 

Adj  R-sq 

0.7967 

C.V. 

8.67421 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter~0 

Prob 

'  >  1 T I 

INTERCEP 

1 

135.335938 

1.03761931 

130.429 

0.0001 

B1 

1 

8.101563 

2.74528266 

2.951 

0.0041 

B2 

1 

2.164063 

2.74528266 

0.788 

0.4327 

B3 

i 

2.226563 

2.74528266 

0.811 

0.4196 

B4 

1 

-8.710938 

2.74528266 

-3.173 

0.0021 

B5 

1 

-6.585938 

2.74528266 

-2.399 

0.0186 

B6 

1 

0.351563 

2.74528266 

0.128 

0.8984 

B7 

1 

0.789063 

2.74528266 

0.287 

0.7745 

B 

1 

20.960938 

1.03761931 

20.201 

0.0001 

H 

1 

5.476563 

1.03761931 

5.278 

0.0001 

BJ 

1 

-2.539063 

1.03761931 

-2.447 

0.0164 

FH 

1 

2.382813 

1.03761931 

2.296 

0.0241 

BF 

1 

2.367188 

1.03761931 

2.281 

0.0250 

CE 

1 

2.054688 

1.03761931 

1.980 

0.0508 

HJ 

1 

-2.023438 

1.03761931 

-1.950 

0.0544 

DH 

1 

2.023438 

1.03761931 

1.950 

0.0544 

DJ 

1 

1.945313 

1.03761931 

1.875 

0.0642 

CG 

1 

-1.914063 

1.03761931 

-1.845 

0.0685 

G 

1 

1.898438 

1.03761931 

1.830 

0.0707 

EG 

1 

-1.820313 

1.03761931 

-1.754 

0.0829 

JK 

1 

1.820313 

1.03761931 

1.754 

0.0829 

AH 

1 

-1.632813 

1.03761931 

-1.574 

0.1192 

BE 

1 

1.585938 

1.03761931 

1.528 

0.1300 

J 

1 

-1.539063 

1.03761931 

-1.483 

0.1416 

AF 

1 

-1.523438 

1.03761931 

-1.468 

0.1457 

GH 

1 

1.507813 

1.03761931 

1.453 

0.1498 

BG 

1 

1.492188 

1.03761931 

1.438 

0.1540 

BH 

1 

1.445313 

1.03761931 

1.393 

0.1672 

EJ 

1 

-1.414063 

1.03761931 

-1.363 

0.1765 

DK 

1 

1.398438 

1.03761931 

1.348 

0.1812 

FJ 

1 

-1.382813 

1.03761931 

-1.333 

0.1861 

EK 

1 

1.351563 

1.03761931 

1.303 

0.1962 

CD 

1 

-1.304688 

1.03761931 

-1.257 

0.2120 

AJ 

1 

-1.273438 

1.03761931 

-1.227 

0.2230 

F 

1 

1.273438 

1.03761931 

1.227 

0.2230 

FK 

1 

-1.273438 

1.03761931 

-1.227 

0.2230 

AE 

1 

1.164063 

1.03761931 

1.122 

0.2650 

GK 

1 

-1.148438 

1.03761931 

-1.107 

0.2714 

GJ 

1 

1.117188 

1.03761931 

1.077 

0.2846 

AB 

1 

-1.117188 

1.03761931 

-1.077 

0.2846 

EH 

1 

1.039063 

1.03761931 

1.001 

0.3194 

B-17 


No-Attack  Case  Max  Adj  Raqr  Day  18 
Analysis  o f  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

Error 

C  Total 

33  59024.25781  1788.61387 

94  11037.23438  117.41739 

127  70061.49219 

15.233 

0.0001 

Root  MSE 
Dep  Mean 

C.V. 

10.83593  R-square 

126.24219  Adj  R-aq 

8.58345 

0.8425 

0.7872 

M 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

126.242188 

0.95776998 

131.808 

0.0001 

81 

1 

9.320313 

2.53402118 

3.678 

0.0004 

82 

1 

-2.929688 

2.53402118 

-1.156 

0.2506 

83 

1 

-2.492188 

2.53402118 

-0.983 

0.3279 

B4 

1 

-3.117188 

2.53402118 

-1.230 

0.2217 

B5 

1 

-2.179688 

2.53402118 

-0.860 

0.3919 

B6 

1 

0.007813 

2.53402118 

0.003 

0.9975 

B7 

1 

0.070313 

2.53402118 

0.028 

0.9779 

B 

1 

18.273438 

0.95776998 

19.079 

0.0001 

BE 

1 

3.976563 

0.95776998 

4.152 

0.0001 

BC 

1 

3.304688 

0.95776998 

3.450 

0.0008 

CF 

1 

-2.898438 

0.95776998 

-3.026 

0.0032 

F 

1 

2.820313 

0.95776998 

2.945 

0.0041 

H 

1 

2.726563 

0.95776998 

2.847 

0.0954 

BF 

1 

2.601563 

0.95776998 

2.716 

0.0079 

GJ 

1 

2.523438 

0.95776998 

2.635 

0.0098 

FK 

1 

-2.382813 

0.95776998 

-2.488 

0.0146 

HJ 

1 

-2.304688 

0.95776998 

-2.406 

0.0181 

EF 

1 

-2.195313 

0.95776998 

-2.292 

0.0241 

BG 

1 

2.023438 

0.95776998 

2.113 

0.0373 

AE 

1 

1.914063 

0.95776998 

1.998 

0.0486 

CK 

1 

-1.867188 

0.95776998 

-1.950 

0.0542 

GH 

1 

1.757813 

0.95776998 

1.835 

0.0696 

EH 

1 

1.742188 

0.95776998 

1.819 

0.0721 

AK 

1 

1.601563 

0.95776998 

1.672 

0.0978 

BD 

1 

1.554688 

0.95776998 

1.623 

0.1079 

AD 

1 

-1.382813 

0.95776998 

-1.444 

0.1521 

J 

1 

-1.226563 

0.95776998 

-1.281 

0.2035 

JK 

1 

1.226563 

0.95776998 

1.281 

0.2035 

DK 

1 

1.195313 

0.95776998 

1.248 

0.2151 

DE 

1 

-1.148438 

0.95776998 

-1.199 

0.2335 

AJ 

1 

1.023438 

0.95776998 

1.069 

0.2880 

EG 

1 

-0.992188 

0.95776998 

-1.036 

0.3029 

CJ 

1 

-0.976563 

0.95776998 

-1.020 

0.3105 

B-18 


No-Attack  Caae  Max  Ad j  Raqr  Day  19 
Analysis  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Model 

36  72925.03125  2025.69531 

6.608 

Error 

91  27895.43750  306.54327 

C  Total 

127  100820.46875 

Root  MSE 

17.50838  R-square 

0.7233 

Dep  Mean 

115.10938  Adj  R-sq 

0.6139 

C.V. 

15.21021 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob 

XNTERCEP 

1 

115.109375 

1.54753652 

74.382 

0 

B1 

1 

11.640625 

4.09439679 

2.843 

0 

B2 

1 

-5.546875 

4.09439679 

-1.355 

0 

B3 

1 

3.203125 

4.09439679 

0.782 

0 

B4 

1 

-12.609375 

4.09439679 

-3.080 

0 

B5 

1 

8.390625 

4.09439679 

2.049 

0 

B6 

1 

-2.296875 

4.09439679 

-0.561 

0 

B7 

1 

-4.921875 

4.09439679 

-1.202 

0 

B 

1 

12.656250 

1.54753652 

8.178 

0 

E 

1 

-6.031250 

1.54753652 

-3.897 

0 

J 

1 

-5.640625 

1.54753652 

-3.645 

0 

BK 

1 

5.593750 

1.54753652 

3.615 

0 

HK 

1 

5.421875 

1.54753652 

3.504 

0 

JK 

1 

4.609375 

1.54753652 

2.979 

0 

EJ 

1 

-4.562500 

1.54753652 

-2.948 

0 

BC 

1 

4.328125 

1.54753652 

2.797 

0 

K 

1 

4.265625 

1.54753652 

2.756 

0 

CE 

1 

4.234375 

1.54753652 

2.736 

0 

DK 

1 

4.062500 

1.54753652 

2.625 

0 

GJ 

1 

3.937500 

1.54753652 

2.544 

0 

EK 

1 

3.718750 

1.54753652 

2.403 

0 

C 

1 

3.093750 

1.54753652 

1.999 

0 

CK 

1 

-2.968750 

1.54753652 

-1.918 

0 

EH 

1 

-2.750000 

1.54753652 

-1.777 

0 

HJ 

1 

-2.734375 

1.54753652 

-1.767 

0 

AC 

1 

-2.640625 

1.54753652 

-1.706 

0 

D 

1 

-2.625000 

1.54753652 

-1.696 

0 

BJ 

1 

-2.375000 

1.54753652 

-1.535 

0 

DH 

1 

-2.375000 

1.54753652 

-1.535 

0 

GK 

1 

1.843750 

1.54753652 

1.191 

0 

DE 

1 

-1.828125 

1.54753652 

-1.181 

0 

AG 

1 

-1.796875 

1.54753652 

-1.161 

0 

CF 

1 

1.796875 

1.54753652 

1.161 

0 

EG 

1 

-1.734375 

1.54753652 

-1.121 

0 

BE 

1 

-1.703125 

1.54753652 

-1.101 

0 

BH 

1 

-1.625000 

1.54753652 

-1.050 

0 

BF 

1 

1.609375 

1.54753652 

1.040 

0 

Prob>F 

0.0001 


>  I T I 

.0001 
.0055 
.1789 
.  4381 
.0027 
.0433 
.5762 
.2324 
.0001 
.0002 
.0004 
.0005 
.0007 
.0037 
.0041 
.0063 
.0071 
.0075 
.0102 
.0126 
.0183 
.0486 
.0582 
.0789 
.0806 
.0914 
.0933 
.1283 
.1283 
.2366 
.2406 
.2486 
.2486 
.2654 
.2740 
.2965 
.3011 


B-19 


No-Attack  Caa*  Max  Adj  Rsqr  Day  20 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares 

Square 

F  Value 

Model 

36  50581.93750 

1405.05382 

7.845 

Error 

91  16298.03125 

179.09924 

C  Total 

127  66879.96875 

Root  MSE 

13.38280  ! 

R-square 

0.7563 

Dep  Mean 

109.01563 

Adj  R-sq 

0.6599 

C.V. 

12.27604 

Parameter 

Estimates 

Variable  DF 


INTERCEP  1 
B1  1 

B2  1 

B3  1 

B4  1 

B5  1 

B6  1 

B7  1 

B  1 

K  1 

BK  1 

CE  1 

J  1 

DG  1 

BD  1 

D  1 

JK  1 

DE  1 

EJ  1 

E  1 

F  1 

EK  1 

BJ  1 

FJ  1 

BG  1 

H  1 

AC  1 

FH  1 

GH  1 

A  1 

CG  1 

CK  1 

FG  1 

AE  1 

BH  1 

DF  1 

FK  1 


Parameter 

Estimate 

109.015625 

1.421875 

4.109375 

1.546875 

-7.640625 

7.171875 
-1.203125 
-8.140625 
12.062500 

5.218750 

4.765625 

4.578125 

-4.156250 

-3.656250 

3.515625 

3.312500 

3.171875 
-3.171875 
-3.109375 
-2 . 875000 
-2.750000 

2.703125 

-2.515625 

2.390625 

2.281250 
2.000000 
1.968750 

-1.921875 

1.718750 

1.671875 

1.343750 

-1.328125 

-1.312500 

1.281250 
-1.265625 
-1.234375 
-1.203125 


Standard 

Error 


1.18288328 

3.12961498 

3.12961498 

3.12961498 

3.12961498 

3.12961498 

3.12961498 

3.12961498 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 

1.18288328 


T  for  HO: 
Parameter-0 

92.161 

0.454 

1.313 

0.494 

-2.441 

2.292 

-0.384 

-2.601 

10.198 

4.412 
4.029 
3.870 

-3.514 
-3.091 
2.972 
2.800 
2.681 
-2 . 681 
-2.629 
-2.431 
-2.325 
2.285 
-2.127 
2.021 
1.929 
1.691 
1.664 
-1.625 
1.453 

1.413 
1.136 

-1.123 

-1.110 

1.083 

-1.070 

-1.044 

-1.017 


Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

c 

c 

c 

0 

0 

c 

c 

c 


Prob>F 

0.0001 


>  IT  | 

.0001 

.6507 

.1925 

.6223 

.0166 

.0242 

.7016 

.0108 

.0001 

.0001 

.0001 

.0002 

.0007 

.0026 

.0038 

.0062 

.0087 

.0087 

.0101 

.0170 

.0223 

.0246 

.0362 

.0462 

.0569 

.0943 

.0995 

.1077 

.1497 

.1610 

.2589 

.2645 

.2701 

.2816 

.2875 

.2995 

.3118 


B-20 


No-Attack  Casa  Max  ftdj  Rsqr  Day  21 
Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

39  66789. 

99219  1712. 

56390 

10. 

945 

0.0001 

Error 

88  13769. 

50000  156. 

47159 

C 

Total 

127  80559. 

49219 

Root 

MSE 

12.50886 

R- square 

0.8291 

Dep  Mean 

110.24219 

Adj  R-sq 

0.7533 

C.V. 

11.34671 

Parameter  Estimates 

Parameter 

Standard 

T 

for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

110.242188 

1.10563751 

99.709 

0.0001 

B1 

1 

1.507813 

2.92524189 

0.515 

0.6075 

B2 

1 

4.570313 

2.92524189 

1.562 

0.1218 

B3 

1 

-3.304688 

2.92524189 

-1.130 

0.2617 

B4 

1 

-9.304688 

2.92524189 

-3.181 

0.0020 

B5 

1 

-2.054688 

2.92524189 

-0.702 

0.4843 

B6 

1 

3.445313 

2.92524189 

1.178 

0.2421 

B7 

1 

2.007813 

2.92524189 

0.686 

0.4943 

B 

1 

18.679688 

1.10563751 

16.895 

0.0001 

H 

1 

5.351563 

1.10563751 

4.840 

0.0001 

BE 

1 

3.695313 

1.10563751 

3.342 

0.0012 

FJ 

1 

-3.304688 

1.10563751 

-2.989 

0.0036 

BK 

1 

-3.023438 

1.10563751 

-2.735 

0.0076 

FH 

1 

2.914063 

1.10563751 

2.636 

0.0099 

BH 

1 

2.632813 

1.10563751 

2.381 

0.0194 

K 

1 

-2.273438 

1.10563751 

-2.056 

0.0427 

HK 

1 

-2.257813 

1.10563751 

-2.042 

0.0441 

EH 

1 

2.210938 

1.10563751 

2.000 

0.0486 

CF 

1 

-2.179688 

1.10563751 

-1.971 

0.0518 

HJ 

1 

-2.132813 

1.10563751 

-1.929 

0.0569 

GH 

1 

2.070313 

1.10563751 

1.873 

0.0645 

JK 

1 

2.054688 

1.10563751 

1.858 

0.0665 

AE 

1 

2.023438 

1.10563751 

1.830 

0.0706 

GJ 

1 

1.882813 

1.10563751 

1.703 

0.0921 

G 

1 

1.867188 

1.10563751 

1.689 

0.0948 

BC 

1 

1.789063 

1.10563751 

1.618 

0.1092 

DJ 

1 

1.789063 

1.10563751 

1.618 

0.1092 

BD 

1 

1.773438 

1.10563751 

1.604 

0.1123 

D 

1 

1.679688 

1.10563751 

1.519 

0.1323 

DG 

1 

1.617188 

1.10563751 

1.463 

0.1471 

EK 

1 

-1.570313 

1.10563751 

-1.420 

0.1591 

FG 

1 

-1.414063 

1.10563751 

-1.279 

0.2043 

FK 

1 

-1.398438 

1.10563751 

-1.265 

0.2093 

AD 

1 

-1.367188 

1.10563751 

-1.237 

0.2195 

DE 

1 

-1.304688 

1.10563751 

-1.180 

0.2412 

DH 

1 

1.257813 

1.10563751 

1.138 

0.2584 

EF 

1 

1.257813 

1.10563751 

1.138 

0.2584 

C 

1 

-1.210938 

1.10563751 

-1.095 

0.2764 

F 

1 

1.148438 

1.10563751 

1.039 

0.3018 

AJ 

1 

1.117188 

1.10563751 

1.010 

0.3151 

B-21 


No-Attack  Casa  Max  Adj  Raqr  Day  22 


Analysis  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Model 

31  49053.92969  1582.38483 

10.881 

Error 

96  13960.68750  145.42383 

C  Total 

127  63014.61719 

Root  MSE 

12.05918  R-square 

0.7785 

Dep  Mean 

102.94531  Adj  R-sq 

0.7069 

C.V. 

11.71416 

Parameter  Estimates 


Variable  DF 


INTERCEP  1 
B1  1 

B2  1 

B3  1 

B4  1 

B5  1 

B6  1 

B7  1 

B  1 

H  1 

HJ  1 

K  1 

BG  1 

BK  1 

AK  1 

G  1 

BC  1 

EG  1 

EH  1 

HK  1 

FK  1 

GH  1 

BH  1 

BD  1 

CG  1 

FG  1 

BF  1 

AB  1 

C  1 

E  1 

AF  1 

BJ  1 


Parameter 

Estimate 

102.945313 

0.117188 

1.117188 
-1.132813 
-1.820313 
-2.320313 

1.617188 

0.742188 

16.476563 

4.117188 
-3.789063 
-3.664063 

2.648438 

-2.632813 

2.507813 

2.460938 

1.960938 

1.945313 

1.882813 

-1.773438 

-1.726563 

1.695313 

1.585938 

1.570313 

1.476563 

-1.476563 

1.445313 

1.335938 

-1.257813 

-1.101563 

-1.101563 

-1.085938 


Standard 

Error 

1.06589102 

2.82008255 

2.82008255 

2.82008255 

2.82008255 

2.82008255 

2.82008255 

2.82008255 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1 . 06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 
1.06589102 


T  for  HO: 

Parameter*^ 

96.581 

0.042 

0.396 

-0.402 

-0.645 

-0.823 

0.573 

0.263 

15.458 

3.863 

-3.555 

-3.438 

2.485 

-2.470 

2.353 

2.309 

1.840 

1.825 

1.766 

-1.664 

-1.620 

1.591 

1.488 

1.473 

1.385 

-1.385 

1.356 

1.253 

-1.180 

-1.033 

-1.033 

-1.019 


Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


Prob>F 

0.0001 


>  ITI 

.0001 
.9669 
.6929 
.  6888 
.5202 
.4127 
.5677 
.7930 
.0001 
.0002 
.0006 
.0009 
.0147 
.0153 
.0207 
.0231 
.0689 
.0711 
.0805 
.0994 
.1085 
.1150 
.1401 
.1440 
.1692 
.1692 
.1783 
.2131 
.2409 
.3040 
.3040 
.3109 


B-22 


No- At tack  Case  Max  Adj  Raqr  Day  23 


Analysis  of 

Variance 

Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Prob>F 

Model 

33 

56804.19531 

1721.33925 

11.971 

0.0001 

Error 

94 

13516.92188 

143.79704 

C  Total 

127 

70321.11719 

Root 

MSE 

11.99154 

R-square 

0.8078 

Dep  Mean 

97.30469 

Adj  R-sq 

0.7403 

C.V. 

12.32370 

Parameter  Estimates 

Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter^ 

Prob  >  1 T | 

INTERCEP 

1 

97.304688 

1.05991244 

91.804 

0.0001 

B1 

1 

1.820313 

2.80426473 

0.649 

0.5178 

B2 

1 

4.382813 

2.80426473 

1.563 

0.1214 

B3 

1 

-6.179688 

2.80426473 

-2.204 

0.0300 

B4 

1 

-2.304688 

2.80426473 

-0.822 

0.4132 

B5 

1 

1.820313 

2.80426473 

0.649 

0.5178 

B6 

1 

-0.054687 

2.80426473 

-0.020 

0.9845 

B7 

1 

2.5C7813 

2.30426473 

0.894 

0.3735 

B 

1 

18.00/813 

1.05991244 

16.990 

0.0001 

H 

1 

3.960938 

1.05991244 

3.737 

0.0003 

E 

1 

-3.007813 

1.05991244 

-2.838 

0.0056 

BH 

1 

2.882813 

1.05991244 

2.720 

0.0078 

AC 

1 

2.820313 

1.05991244 

2.661 

0.0092' 

HJ 

1 

-2.789063 

1.05991244 

-2.631 

0.0099 

EK 

1 

-2.460938 

1.05991244 

-2.322 

0.0224 

K 

1 

-2.242188 

1.05991244 

-2.115 

0.0370 

FG 

1 

-2.195313 

1.05991244 

-2.071 

0.0411 

FJ 

1 

-2.179688 

1.05991244 

-2.056 

0.0425 

BC 

1 

1.992188 

1.05991244 

1.880 

0.0633 

EG 

1 

1.976563 

1.05991244 

1.865 

0.0653 

JK 

1 

1.851563 

1.05991244 

1.747 

0.0839 

BG 

1 

1.835938 

1.05991244 

1.732 

0.0865 

GJ 

1 

1.820313 

1.05991244 

1.717 

0.0892 

G 

1 

1.757813 

1.05991244 

1.658 

0.1006 

AE 

1 

1.710938 

1.05991244 

1.614 

0.1098 

F 

1 

1.570313 

1.05991244 

1.482 

0.1418 

AF 

1 

-1.554688 

1.05991244 

-1.467 

0.1458 

DE 

1 

-1.476563 

1.05991244 

-1.393 

0.1669 

FH 

1 

1.382813 

1.05991244 

1.305 

0.1952 

EF 

1 

-1.304688 

1.05991244 

-1.231 

0.2214 

EH 

1 

1.242188 

1.05991244 

1.172 

0.2442 

C 

1 

-1.179688 

1.05991244 

-1.113 

0.2685 

FK 

1 

-1.164063 

1.05991244 

-1.098 

0.2749 

DG 

1 

1.070313 

1.05991244 

1.010 

0.3152 

No-Attack  Case  Max  Adj  Raqr  Day  24 
Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean 

Square 

F  Value 

Prob>F 

Model 

38 

50329.18750 

1324.45230 

9.226 

0.0001 

Error 

C  Total 

89 

127 

12776.31250 

63105.50000 

143.55407 

Root  MSB  11.98141  R-square  0.7975 
Dep  Mean  91.18750  Adj  R-sq  0.7111 
C.V.  13.13931 

Parameter  Estimates 


Variable  DF 


INTERCEP  1 
B1  1 

B2  1 

B3  1 

B4  1 

B5  1 

B6  1 

B7  1 

B  1 

E  1 

FJ  1 

K  1 

AG  1 

CJ  1 

A  1 

J  1 

AH  1 

AB  1 

H  1 

EK  1 

AC  1 

EH  1 

EG  1 

AJ  1 

BH  1 

HJ  1 

EJ  1 

JK  1 

CD  1 

AE  1 

GK  1 

F  1 

BG  1 

FH  1 

BD  1 

FK  1 

DG  1 

CH  1 

GH  1 


Parameter 

Estimate 

91.187500 

-2.000000 

8.062500 

-7.562500 

-6.000000 

-4.250000 

4.250000 

4.312500 

15.718750 

-4.750000 

-3.265625 

-3.187500 

2.484375 

2.281250 

2.218750 

2.125000 

-2.109375 

2.093750 

2.015625 

-1.875000 

1.875000 

1.734375 

1.578125 

-1.562500 

1.515625 

-1.515625 

1.468750 

1.468750 

-1.375000 

1.375000 

1.328125 

1.296875 

-1.265625 

1.187500 

1.187500 

-1.171875 

1.171875 

-1.109375 

-1.062500 


Standard 

Error 

1.05901662 

2.80189460 

2.80189460 

2.80189460 

2.80189460 

2.80189460 

2.80189460 

2.80189460 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 

1.05901662 


T  for  HO: 

Parameter-0 

86.106 

-0.714 

2.878 

-2.699 

-2.141 

-1.517 

1.517 

1.539 

14.843 

-4.485 

-3.084 

-3.010 

2.346 

2.154 

2.095 

2.007 

-1.992 

1.977 

1.903 

-1.771 

1.771 

1.638 

1.490 

-1.475 

1.431 

-1.431 

1.387 

1.387 

-1.298 

1.298 

1.254 

1.225 

-1.195 

1.121 

1.121 

-1.107 

1.107 

-1.048 

-1.003 


Prob  >  IT | 

0.0001 

0.4772 

0.0050 

0.0083 

0.0350 

0.1329 

0.1329 

0.1273 

0.0001 

0.0001 

0.0027 

0.0034 

0.0212 

0.0339 

0.0390 

0.0478 

0.0495 

0.0511 

0.0602 

0.0801 

0.0801 

0.1050 

0.1397 

0.1436 

0.1559 

0.1559 

0.1689 

0.1689 

0.1975 

0.1975 

0.2131 

0.2240 

0.2352 

0.2652 

0.2652 

0.2715 

0.2715 

0.2977 

0.3184 


B-24 


No-Attack  Casa  Max  Ad j  Rsqr  Day  25 
Analysis  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

Error 

C  Total 

29  39123.66406  1349.09186 

98  17117.70313  174.67044 

127  56241.36719 

7.724 

0.0001 

Root  MSE 
Dep  Mean 
C.V. 

13.21629  R-square 
84.57031  Adj  R-sq 
15.62758 

0.6956 

0.6056 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter-0 

Prob  >  1 T I 

INTERCEP 

1 

84.570313 

1.16816643 

72.396 

0.0001 

B1 

1 

-3.882813 

3.09067787 

-1.256 

0.2120 

B2 

1 

0.867188 

3.09067787 

0.281 

0.7796 

B3 

1 

-1.695313 

3.09067787 

-0.549 

0.5846 

B4 

1 

1.179688 

3.09067787 

0.382 

0.7035 

B5 

1 

-5.007813 

3.09067787 

-1.620 

0.1084 

B6 

1 

7.867187 

3.09067787 

2.545 

0.0125 

B7 

1 

0.617188 

3.09067787 

0.200 

0.8421 

B 

1 

12.960938 

1.16816643 

11.095 

0.0001 

E 

1 

-5.039063 

1.16816643 

-4.314 

0.0001 

AC 

1 

4.179688 

1.16816643 

3.578 

0.0005 

BJ 

1 

3.507813 

1.16816643 

3.003 

0.0034 

AE 

1 

3.445313 

1.16816643 

2.949 

0.0040 

K 

1 

-3.117188 

1.16816643 

-2.668 

0.0089 

J 

1 

2.617188 

1.16816643 

2.240 

0.0273 

BE 

1 

-2.148438 

1.16816643 

-1.839 

0.0689 

DH 

1 

2.085938 

1.16816643 

1.786 

0.0772 

FJ 

1 

-1.945313 

1.16816643 

-1.665 

0.0991 

CG 

1 

1.757813 

1.16816643 

1.505 

0.1356 

HJ 

1 

1.742188 

1.16816643 

1.491 

0.1391 

EJ 

1 

1.664063 

1.16816643 

1.425 

0.1575 

BK 

1 

-1.664063 

1.16816643 

-1.425 

0.1575 

CK 

1 

-1.617188 

1.16816643 

-1.384 

0.1694 

C 

1 

-1.585938 

1.16816643 

-1.358 

0.1777 

CJ 

1 

1.585938 

1.16816643 

1.358 

0.1777 

EK 

1 

-1.507813 

1.16816643 

-1.291 

0.1998 

BH 

1 

-1.320313 

1.16816643 

-1.130 

0.2611 

DJ 

1 

-1.273438 

1.16816643 

-1.090 

0.2783 

CH 

1 

-1.242188 

1.16816643 

-1.063 

0.2902 

A 

1 

1.179688 

1.16816643 

1.010 

0.3150 

B-25 


Mo-Attack  Caae  Max  Adj  Rsqr  Day  26 
Analyaia  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

35  44069. 

59375  1259.13125 

5 

.539 

0.0001 

Error 

92  20912. 

37500  227.30842 

C  Total 

127  64981. 

96875 

Root 

MSE 

15.07675 

R- square 

0.6782 

Dep  Mean 

76.98438 

Adj  R-sq 

0.5558 

C.V. 

19.58417 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter^ 

Prob  >  | T | 

INTERCEP 

1 

76.984375 

1.33260912 

57.770 

0.0001 

B1 

1 

-1.546875 

3.52575232 

-0.439 

0.6619 

B2 

1 

7.140625 

3.52575232 

2.025 

0.0457 

B3 

1 

-5.609375 

3.52575232 

-1.591 

0.1150 

B4 

1 

-1.984375 

3.52575232 

-0.563 

0.5749 

B5 

1 

-3.609375 

3.52575232 

-1.024 

0.3087 

B6 

1 

6.015625 

3.52575232 

1.706 

0.0913 

B7 

1 

6.015625 

3.52575232 

1.706 

0.0913 

B 

1 

9.593750 

1.33260912 

7.199 

0.0001 

E 

1 

-8.125000 

1.33260912 

-6.097 

0.0001 

BE 

1 

-5.140625 

1.33260912 

-3.858 

0.0002 

BH 

1 

-4.187500 

1.33260912 

-3.142 

0.0023 

C 

1 

-3.781250 

1.33260912 

-2.837 

0.0056 

BJ 

1 

3.453125 

1.33260912 

2.591 

0.0111 

BC 

1 

-2.953125 

1.33260912 

-2.216 

0.0292 

H 

1 

-2.546875 

1.33260912 

-1.911 

0.0591 

CJ 

1 

2.453125 

1.33260912 

1.841 

0.0689 

J 

1 

2.406250 

1.33260912 

1.806 

0.0742 

DK 

1 

2.390625 

1.33260912 

1.794 

0.0761 

GH 

1 

-2.281250 

1.33260912 

-1.712 

0.0903 

AK 

1 

2.234375 

1.33260912 

1.677 

0.0970 

FG 

1 

2.140625 

1.33260912 

1.606 

0.1116 

BK 

1 

2.125000 

1.33260912 

1.595 

0.1142 

DJ 

1 

-2.125000 

1.33260912 

-1.595 

0.1142 

AJ 

1 

-2.062500 

1.33260912 

-1.548 

0.1251 

CD 

1 

-1.937500 

1.33260912 

-1.454 

0.1494 

DE 

1 

-1.875000 

1.33260912 

-1.407 

0.1628 

AE 

1 

1.718750 

1.33260912 

1.290 

0.2004 

BD 

1 

-1.656250 

1.33260912 

-1.243 

0.2171 

AC 

1 

1.625000 

1.33260912 

1.219 

0.2258 

EF 

1 

-1.609375 

1.33260912 

-1.208 

0.2303 

BG 

1 

-1.546875 

1.33260912 

-1.161 

0.2487 

EJ 

1 

1.453125 

1.33260912 

1.090 

0.2784 

D 

1 

-1.390625 

1.33260912 

-1.044 

0.2994 

AG 

1 

-1.375000 

1.33260912 

-1.032 

0.3049 

JK 

1 

1.343750 

1.33260912 

1.008 

0.3159 

B-26 


No-Attack  Case  Max  Adj  Raqr  Day  27 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF  Squares 

Square 

F  Value 

Prob>F 

Model 

36  50841.21875 

1412.25608 

3.819 

0.0001 

Error 

91  33653.65625 

' 369.82040 

C  Total 

127  84494.87500 

Root  MSE 

19.23071  R- 

■square 

0.6017 

Dep  Mean 

64.90625  Adj  R-sq 

0.4441 

C.V. 

29.62845 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameters 

Prob  >  1 T | 

INTERCEP 

1 

64.906250 

1.69977112 

38.185 

0.0001 

B1 

1 

-6.781250 

4.49717167 

-1.508 

0.1350 

B2 

1 

1.406250 

4.49717167 

0.313 

0.7552 

B3 

1 

0.593750 

4.49717167 

0.132 

0.8953 

B4 

1 

2.968750 

4.49717167 

0.660 

0.5108 

B5 

1 

-2.656250 

4.49717167 

-0.591 

0.5562 

B6 

1 

4.968750 

4.49717167 

1.105 

0.2721 

B7 

1 

4.843750 

4.49717167 

1.077 

0.2843 

BK 

1 

7.671875 

1.69977112 

4.513 

0.0001 

E 

1 

-7.500000 

1.69977112 

-4.412 

0.0001 

BH 

1 

-6.328125 

1.69977112 

-3.723 

0.0003 

K 

1 

5.281250 

1.69977112 

3.107 

0.0025 

DE 

1 

-4.187500 

1.69977112 

-2.464 

0.0156 

J 

1 

4.171875 

1.69977112 

2.454 

0.0160 

BJ 

1 

3.406250 

1.69977112 

2.004 

0.0480 

C 

1 

-3.281250 

1.69977112 

-1.930 

0.0567 

AG 

1 

-3.281250 

1.69977112 

-1.930 

0.0567 

CJ 

1 

3.265625 

1.69977112 

1.921 

0.0578 

H 

1 

-3.218750 

1.69977112 

-1.894 

0.0615 

HJ 

1 

3.203125 

1.69977112 

1.884 

0.0627 

F 

1 

-3.171875 

1.69977112 

-1.866 

0.0653 

GH 

1 

-3.078125 

1.69977112 

-1.811 

0.0735 

CE 

1 

3.000000 

1.69977112 

1.765 

0.0809 

BE 

1 

-2.828125 

1.69977112 

-1.664 

0.0996 

AE 

1 

2.796875 

1.69977112 

1.645 

0.1033 

EF 

1 

-2.640625 

1.69977112 

-1.554 

0.1238 

AK 

1 

2.609375 

1.69977112 

1.535 

0.1282 

GJ 

1 

-2.468750 

1.69977112 

-1.452 

0.1498 

CG 

1 

2.078125 

1.69977112 

1.223 

0.2246 

AC 

1 

2.015625 

1.69977112 

1.186 

0.2388 

CF 

1 

1.984375 

1.69977112 

1.167 

0.2461 

DF 

1 

1.984375 

1.69977112 

1.167 

0.2461 

FH 

1 

-1.984375 

1.69977112 

-1.167 

0.2461 

BF 

1 

-1.843750 

1.69977112 

-1.085 

0.2809 

JK 

1 

1.796875 

1.69977112 

1.057 

0.2933 

BC 

1 

-1.765625 

1.69977112 

-1.039 

0.3017 

DJ 

1 

-1.734375 

1.69977112 

-1.020 

0.3103 
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No-Attack  Casa  Max  Adj  Raqr  Day  28 


Analysis  of  Variance 


Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Prob>F 

Model 

33 

65266.31250 

1977.76705 

6.025 

0.0001 

Error 

94 

30854.56250 

328.24003 

C  Total 

127 

96120.87500 

Root  MSB  18.11740  R-square  0.6790 
Dap  Mean  54.59375  Adj  R-sq  0.5663 
C.V.  33.18584 


Parameter  Estimates 


Parameter 

Standard 

Variable 

DF 

Estimate 

Error 

1NTERCEP 

1 

54.593750 

1.60136667 

B1 

1 

-6.531250 

4.23681796 

B2 

1 

4.843750 

4.23681796 

B3 

1 

0.468750 

4.23681796 

B4 

1 

1.906250 

4.23681796 

B5 

1 

1.718750 

4.23681796 

B6 

1 

-1.968750 

4.23681796 

B7 

1 

2.718750 

4.23681796 

BK 

1 

12.265625 

1.60136667 

K 

1 

11.015625 

1.60136667 

B 

1 

-5.187500 

1.60136667 

OK 

1 

4.937500 

1.60136667 

BH 

1 

-4.515625 

1.60136667 

E 

1 

-3.890625 

1.60136667 

HO 

1 

3.812500 

1.60136667 

AF 

1 

-3.562500 

1.60136667 

CJ 

1 

3.546875 

1.60136667 

H 

1 

-3.421875 

1.60136667 

AG 

1 

-3.328125 

1.60136667 

DF 

1 

3.078125 

1.60136667 

BC 

1 

-2.968750 

1.60136667 

EK 

1 

-2.593750 

1.60136667 

EH 

1 

-2.437500 

1.60136667 

GH 

1 

-2.328125 

1.60136667 

GJ 

1 

-2.296875 

1.60136667 

CK 

1 

-2.265625 

1.60136667 

FK 

1 

2.156250 

1.60136667 

CE 

1 

2.078125 

1.60136667 

G 

1 

-2.031250 

1.60136667 

J 

1 

1.953125 

1.60136667 

A 

1 

1.921875 

1.60136667 

AH 

1 

1.875000 

1.60136667 

FH 

1 

-1.781250 

1.60136667 

AE 

1 

1.687500 

1.60136667 

T  for  HO: 


Parameter*^ 

Prob  >  I T  | 

34.092 

0.0001 

-1.542 

0.1265 

1.143 

0.2558 

0.111 

0.9121 

0.450 

0.6538 

0.406 

0.6859 

-0.465 

0.6432 

0.642 

0.5226 

7.659 

0.0001 

6.879 

0.0001 

-3.239 

0.0017 

3.083 

0.0027 

-2.820 

0.0059 

-2.430 

0.0170 

2.381 

0.0193 

-2.225 

0.0285 

2.215 

0.0292 

-2.137 

0.0352 

-2.078 

0.0404 

1.922 

0.0576 

-1.854 

0.0669 

-1.620 

0.1086 

-1.522 

0.1313 

-1.454 

0.1493 

-1.434 

0.1548 

-1.415 

0.1604 

1.347 

0.1814 

1.298 

0.1976 

-1.268 

0.2078 

1.220 

0.2256 

1.200 

0.2331 

1.171 

0.2446 

-1.112 

0.2688 

1.054 

0.2947 

* 
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No-Attack  Case  Max  Adj  Rsqr  Day  29 
Analysis  of  Variance 


Sum  of  Mean 

Source 

DF  Squares  Square 

F  Value 

Prob>F 

Model 

33  68514. 

88281  2076.20857 

8 

.060 

0.0001 

Error 

94  24213. 

35937  257.58893 

C 

Total 

127  92728. 

24219 

Root 

MSE 

16.04958 

R- aqua re 

0.7389 

Dep  Mean 

49.38281 

Adj  R-sq 

0.6472 

C.V. 

32.50033 

Parameter  Estimates 

Parameter 

Standard  T 

for  HO: 

Variable 

DF 

Estimate 

Error  Parameter-0 

Prob  >  | T | 

INTERCEP 

1 

49.382813 

1.41859561 

34.811 

0.0001 

B1 

1 

-7.570313 

3.75325120 

-2.017 

0.0465 

B2 

1 

1.929688 

3.75325120 

0.514 

0.6084 

B3 

1 

-0.070313 

3.75325120 

-0.019 

0.9851 

B4 

1 

3.304688 

3.75325120 

0.880 

0.3808 

B5 

1 

7.429688 

3.75325120 

1.980 

0.0507 

B6 

1 

-1.132813 

3.75325120 

-0.302 

0.7635 

B7 

1 

-0.445313 

3.75325120 

-0.119 

0.9058 

BK 

1 

12.179688 

1.41859561 

8.586 

0.0001 

K 

1 

11.539063 

1.41859561 

8.134 

0.0001 

B 

1 

-8.570313 

1.41859561 

-6.041 

0.0001 

JK 

1 

5.539063 

1.41859561 

3.905 

0.0002 

BH 

1 

-3.804688 

1.41859561 

-2 . 682 

0.0086 

AG 

1 

-3.445313 

1.41859561 

-2.429 

0.0171 

E 

1 

-3.414063 

1.41859561 

-2.407 

0.0181 

FK 

1 

2.804688 

1.41859561 

1.977 

0.0510 

DK 

1 

2.710938 

1.41859561 

1.911 

0.0591 

HJ 

1 

2.679688 

1.41859561 

1.889 

0.0620 

H 

1 

-2.632813 

1.41859561 

-1.856 

0.0666 

J 

1 

2.570313 

1.41859561 

1.812 

0.0732 

C 

1 

-2.523438 

1.41859561 

-1.779 

0.0785 

EK 

1 

-2.320313 

1.41859561 

-1.636 

0.1053 

CJ 

1 

2.257813 

1.41859561 

1.592 

0.1148 

A 

1 

2.210938 

1.41859561 

1.559 

0.1225 

CK 

1 

-2.148438 

1.41859561 

-1.514 

0.1333 

BJ 

1 

2.148438 

1.41859561 

1.514 

0.1333 

AD 

1 

2.101563 

1.41859561 

1.481 

0.1418 

BC 

1 

-2.007813 

1.41859561 

-1.415 

0.1603 

DF 

1 

1.976563 

1.41859561 

1.393 

0.1668 

CG 

1 

1.945313 

1.41859561 

1.371 

0.1735 

AH 

1 

1.726563 

1.41859561 

1.217 

0.2266 

DE 

1 

1.695313 

1.41859561 

1.195 

0.2351 

G 

1 

-1.648438 

1.41859561 

-1.162 

0.2482 

DG 

1 

-1.570313 

1.41859561 

-1.107 

0.2711 

Mo-Attack  Case  Max  Ad j  Rsqr  Day  30 
Analyaia  of  Variance 


Source 

Sum  of  Mean 

DF  Squares  Square 

F  Value 

Model 

28  67797.15625  2421.32701 

8.837 

Error 

99  27127.27344  274.01286 

C  Total 

127  94924.42969 

Root"  MSE 

16.55333  R-square 

0.7142 

Dep  Mean 

45.22656  Adj  R-sq 

0.6334 

C.V. 

36.60091 

Parameter  Estimates 


Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter»0 

INTERCEP 

1 

45.226563 

1.46312183 

30.911 

B1 

1 

-5.289063 

3.87105650 

-1.366 

B2 

1 

-1.039063 

3.87105650 

-0.268 

B3 

1 

0.648438 

3.87105650 

0.168 

B4 

1 

5.085938 

3.87105650 

1.314 

B5 

1 

7.023438 

3.87105650 

1.814 

B6 

1 

-1.226562 

3.87105650 

-0.317 

B7 

1 

-1.476563 

3.87105650 

-0.381 

K 

1 

11.617188 

1.46312183 

7.940 

BK 

1 

11.085938 

1.46312183 

7.577 

B 

1 

-9.617188 

1.46312183 

-6.573 

JK 

1 

6.617188 

1.46312183 

4.523 

H 

1 

-4.023438 

1.46312183 

-2.750 

HJ 

1 

3.820313 

1.46312183 

2.611 

BJ 

1 

3.726563 

1.46312183 

2.547 

BH 

1 

-3.429688 

1.46312183 

-2.344 

J 

1 

3.414063 

1.46312183 

2.333 

E 

1 

-3.132813 

1.46312183 

-2.141 

EK 

1 

-2.585938 

1.46312183 

-1.767 

A 

1 

2.367188 

1.46312183 

1.618 

FK 

1 

2.164063 

1.46312183 

1.479 

CG 

1 

1.992188 

1.46312183 

1.362 

CJ 

1 

1.992188 

1.46312183 

1.362 

BF 

1 

-1.945313 

1.46312183 

-1.330 

GJ 

1 

-1.773438 

1.46312183 

-1.212 

G 

1 

-1.742188 

1.46312183 

-1.191 

AG 

1 

-1.695313 

1.46312183 

-1.159 

BD 

1 

-1.554688 

1.46312183 

-1.063 

GH 

1 

1.476563 

1.46312183 

1.009 

Prob 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

c 

c 

c 

c 

c 

c 


Prob>F 

0.0001 


>  I  T  | 

.0001 
.1749 
.7889 
.8673 
.1919 
.0727 
.7520 
.7037 
.0001 
.0001 
.0001 
.0001 
i . 0071 
.0104 
.0124 
.0211 
.0216 
.0347 
.0802 
.1089 
.1423 
.1764 
.1764 
.1867 
.2284 
.2366 
.2494 
.2906 
.3153 
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Table  C.2  Daily  Metamodels,  Attack  Case,  Alpha  =  0.05 
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Table  C.3  Daily  Metamodels,  Attack  Case,  Alpha  =  0.01 


Table  C.3  Continued 


Table  C.4  Blocking  Effects,  All  Attack  Case  Models 


Day 

B1 

B2 

B3 

B4 

B5 

B6 

B7 

-10.2 

3.5 

-13.4 

2.8 

16.8 

8.7 

-2.6 

-6.5 

57.3 

-29.4 

6.0 

46.4 

-16.6 

-35.4 

-9.8 

20.3 

-23.6 

28.3 

7.1 

63.9 

-34.4 

3.7 

0.3 

29.7 

9.4 

17.1 

-12.8 

-12.9 

5 

7.1 

-22.4 

53.3 

4.8 

-1.8 

2.7 

-11.8 

6 

-30.9 

9.7 

16.0 

11.3 

14.8 

-8.5 

-26.6 

7 

-26.7 

5.3 

23.0 

8.5 

17.4 

-6.1 

-37.9 

8 

-22.2 

-0.1 

14.5 

5.8 

14.5 

-7.8 

-24.9 

9 

-21.5 

-0.5 

17.5 

11.5 

16.2 

-11.7 

-31.0 

10 

-20.0 

2.4 

21.5 

4.3 

14.3 

-6.0 

-32.1 

11 

-16.3 

-1.2 

14.4 

1.6 

17.1 

-30.2 

12 

-16.9 

-2.9 

17.8 

16.2 

-33.1 

13 

-18.8 

3.1 

15.6 

13.0 

-36.7 

14 

-18.1 

-0.5 

17.0 

10.3 

16.2 

-10.4 

-37.3 

15 

-19.8 

2.4 

12.0 

12.8 

19.2 

-10.8 

-34.0 

16 

-22.1 

0.0 

11.0 

10.9 

18.5 

-7.9 

-30.9 

17 

-17.4 

1.1 

10.0 

8.5 

19.0 

-5.5 

-35.6 

18 

-19.9 

0.5 

6.7 

4.7 

15.0 

-6.1 

-26.3 

19 

-19.2 

-2.1 

10.3 

7.9 

17.2 

-5.6 

-29.1 

20 

-20.9 

-0.9 

11.1 

7.6 

15.0 

-27.9 

21 

1.3 

11.4 

9.0 

12.7 

-29.4 

22 

-18.0 

-1.6 

11.1 

10.5 

8.3 

-24.8 

23 

-21.1 

-0.6 

10.7 

7.3 

12.6 

-22.9 

24 

-18.7 

-6.2 

17.1 

8.7 

10.0 

-4.3 

-23.5 

25 

-17.3 

B| 

11.3 

11.6 

11.0 

-4.3 

-24.7 

26 

-19.7 

1 

12.5 

9.7 

12.0 

-2.5 

-22.0 

27 

-19.3 

■sfl 

12.8 

10.1 

7.6 

1.1 

-22.2 

28 

-19.0 

-0.7 

10.3 

6.5 

7.3 

2.0 

-21.7 

29 

-17.4 

-1.0 

8.4 

9.7 

8.0 

0.8 

-20.1 

30 

-16.5 

-1.7 

9.6 

8.2 

8.3 

-2.2 

•171 

C-10 


Table  D.l  Daily  Metamodels,  No-Attack  Case,  Alpha  -  0.10 


dix  D:  Daily  Mata  models.  No- Attack  Case 


Table  D.l  Continued 
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Table  D.2  Daily  Metamodels,  No-Attack  Case,  Alpha  =  0.05 
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Table  D.3  Daily  Metamodels,  No-Attack  Case,  Alpha  =  0.01 
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Table  D.3  Continued 


12.3 

12.2 

11.1 


3  Continued 


Table  D.4  Blocking  Effects,  All  No-Attack  Case  Models 


Table  E.l  Tabulated  Regression  Data,  Day  One,  No- Attack  Case 
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Note®:  MSB  and  common  standard  error  are  for  a  model  of  the  sixe  indicated  by  "f  in  model’,  including  blocking  terms  but  not  the  interoept. 
ITI  for  last  variable  is  the  absolute  value  of  the  T  statistic  for  a  variable  when  it  is  the  last  variable  included  in  a  model. 


Table  E.2  Probabilities  for  t  Statistic,  Day  One,  No-Attack 


Number  of  Variables 

ITI 

Probability  >  1 T 1 

in  Model 

8 

3.2323 

0.00159 

9 

2.9138 

0.00427 

10 

2.7804 

11 

2.0925 

0.03857 

12 

1.9710 

0.05113 

13 

1.9578 

0.05270 

14 

1.6725 

0.09719 

15 

1.6473 

0.10230 

16 

1.5431 

0.12565 

17 

1.5529 

0.12332  | 

18 

1.4842 

0.14064  | 

19 

1.4530 

0.14912  1 

20 

1.4210 

0.15822  I 

21 

1.1897 

0.23682  § 

22 

1.1920 

0.23595  | 

23 

1.1152 

0.26734  1 

24 

1.0769 

0.28404  I 

25 

1.0381 

0.30168  1 

26 

0.9594 

0.33965  I 

27 

0.9195 

0.36005  1 
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Table  F.l  Daily  Metamodels,  Attack  Case,  Subjective  Selection 


Pill  interaction  wiUi:  ABDR  Interaction  with:  Recov  interaction  with: 

ABDR  Recov  Pen  A  IS  Spt  Eq  Spares  Miss  Fuel  Recov  Pen  A  IS  Spt  Eq  Spares  Miss  Fuel  Pen  AIS  Spt  Eq  Spares  Mias  Fuel 

BC  BD  BE  BF  BG  BH  BJ  BK  CD  CE  CF  CG  CH  CJ  CK  DE  DF  DG  DH  DJ  DK 


Table  G.l  Daily  Metamodels,  No-Attack  Case,  Subjective  Selection 


Table  G.l  Continued 


12.3 

12.2 

11.1 


Table  G.l  Continued 
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Attack  Case,  Day  2 
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Regression  Coefficient 


Percent 
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Regression  Coefficient 


Partial  R-Squared,  Percent 


Attack  Case,  Day  6 
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Attack  Case,  Day  8 
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Attack  Case,  Day  9 
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Attack  Case,  Day  10 
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Attack  Case,  Day  12 
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Attack  Case,  Day  15 
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Regression  Coefficient 
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Attack  Case,  Day  17 
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Regression  Coefficient 


Partial  R-Squared,  Percent 


Attack  Case,  Day  18 


Partial  R-Squarod  [^}  Regression  Coefficient 
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Attack  Case,  Day  19 
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Regression  Coefficient 


Partial  R-Squared,  Percent 
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Regression  Coefficient 


Attack  Case,  Day  27 
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Attack  Case,  Day  28 
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Regression  Coefficient 


Table  H.l  Intercept  Parameters,  Attack  Case 


Day 

Intercept  Parameter 

1 

89.6 

2 

83.2 

3 

104.3 

4 

92.3 

5 

102.1 

6 

166.2 

7 

148.6 

8 

145.7 

9 

137.3 

10 

132.3 

11 

127.1 

12 

121.2 

13 

116.5 

14 

110.3 

15 

106.8 

16 

100.0 

17 

95.9 

18 

89.0  | 

19 

86.5 

20 

79.2 

21 

74.5 

22 

69.6 

23 

65.7 

24 

62.1 

25 

59.0 

26 

55.6 

27 

51.6 

28 

48.7 

29 

46.6 

30 

42.9 
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Percent 


No-Attack  Case,  Day  26 
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Table  1.1  Intercept  Parameters,  No- Attack  Case 


Day 

Intercept  Parameter 

1 

264.2 

2 

212.9 

3 

210.0  f 

4 

205.2 

5 

198.1  1 

6 

188.5 

7 

185.5 

8 

171.4 

9 

151.1  ] 

10 

168.0 

11 

173.5 

12 

166.8 

13 

160.4 

14 

153.8 

15 

144.3 

16 

140.2 

17 

135.3 

18 

126.2 

19 

115.1 

20 

a09.0 

21 

110.2 

22 

102.9 

23 

97.3 

24 

91.2 

25 

!  84.6  | 

26 

77.0 

27 

64.9 

28 

54.6 

29 

49.4 

30 

45.2 

1-31 
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